How Datamine works

1. User Input: When you enter a URL into Datamine, the application starts by fetching the Terms of Service (ToS) and Privacy Policy links associated with that URL.
2. Fetching Content: For each identified ToS and Privacy Policy URL, the application retrieves the textual content of those pages.
3. Analysis:
   • AI Ops: Application sends a request to Google's Text-Bison 8k model. Keeping only the first 30k characters. This satsifies many terms. We skip RAG here for simplicity, directness and contorl.
   • AI: Text-Bison assesses the provided terms given the following instructions.
     • Prompt:

       
       Carefully study the following terms or privacy policy. Using this context, answer the following questions. When making assumptions, clearly state them in your analysis. If no data are provided say so.  You must answer in complete sentences.
       
       - `license`:  Does the company have an irrevocable, worldwide, non-exclusive, or royalty-free license to Customer Data? Or does it not? Answer in one or two sentences. 
       
       - `is_moat`: Is the company using the aggregation of customer data as a means to create a competitive advantage or barrier for competitors (commonly referred to as a "competitive moat")? Does the document suggest that the company is aggregating customer data with the intention of leveraging it as a significant competitive asset? Provide direct quotations or paragraph numbers from the document that support your claim. Answer this question in string format in 2-3 sentences.
       
       Provide your answers in json format.  
       
       privacy policy and terms:
       
             terms: {terms}
           

     • Model Configuration:

       
           parameters = {
               "max_output_tokens": 1000,
               "temperature": 0.7,
               "top_p": 0.8,
               "top_k": 40
           }
           model = TextGenerationModel.from_pretrained("text-bison")
                     

4. Displaying Results: The analysis results are then sent back to the application and displayed to you. You'll see details about the license and whether the company in question uses data as a competitive moat.
5. Transparent and Informed: All the processes, from fetching the content to analyzing it, are automated and rely on predefined logic and external cloud functions. This ensures a consistent and transparent analysis for every URL you submit.