Introduction

​

Tchunky is a service for analysis and chunking to convert long-form text documents into RAG/LLM-ready data. We use transformer based segmentation models to convert text into chunks enhanced with context from the original document. Chunks also include keywords and timeframe for further semantic search improvements.

The way you use the Tchunky API is by creating a task for each document you want to process, and then polling the task until it is completed.

If you provide an optional webhook URL in your settings, we will send a POST request to that URL every time a chunk is completed. For some users, this is a more convenient way to process chunks into embeddings asynchronously, instead of having to pull all chunks at the same time from the API.

​

Setup

1. Create an Account

   Go to tchunky.com and create an account. Once logged in, navigate to the Settings (gear icon in the top right) to obtain your API key.

2. Create a task

   To create a new chunking task:

   curl -X POST https://tchunky.com/api/v1/tasks \
        -H "Content-Type: application/json" \
        -H "X-API-Key: YOUR_API_KEY" \
        -d '{"content": "[full text of your document here]"}'
   
   # Response:
   # {
   #   "task": {
   #     "id": "tchunky_06aa97c8-7cc0-4191-b",
   #     "status": "pending",
   #     "url": "https://tchunky.com/api/v1/tasks/tchunky_06aa97c8-7cc0-4191-b",
   #     "created_at": "2023-04-20T12:00:00Z"
   #   }
   # }

3. Check the status of a task:

   curl https://tchunky.com/api/v1/tasks/tchunky_06aa97c8-7cc0-4191-b \
        -H "X-API-Key: YOUR_API_KEY"
   
   # Response:
   # {
   #   "task": {
   #     "id": tchunky_06aa97c8-7cc0-4191-b,
   #     "status": "complete",
   #     "chunks_count": 10,
   #     "created_at": "2023-04-20T12:00:00Z",
   #     "updated_at": "2023-04-20T12:05:00Z",
   #     "text": {
   #       "id": 456,
   #       "title": "Example Article",
   #       "category": "Blog Post",
   #       "summary": "A brief summary of the content...",
   #       "chunks_count": 10,
   #       "status": "complete"
   #     },
   #     "sections": [
   #       {
   #         "title": "Introduction",
   #         "status": "complete"
   #       },
   #       {
   #         "title": "Main Content",
   #         "status": "complete"
   #       },
   #       {
   #         "title": "Conclusion",
   #         "status": "complete"
   #       }
   #     ]
   #   }
   # }