{"id":6935,"date":"2024-02-26T18:39:42","date_gmt":"2024-02-26T22:39:42","guid":{"rendered":"https:\/\/www.benzinga.com\/apis\/?p=6935"},"modified":"2024-02-26T18:39:49","modified_gmt":"2024-02-26T22:39:49","slug":"utilizing-the-benzinga-news-archive-to-power-your-llm","status":"publish","type":"post","link":"https:\/\/www.benzinga.com\/apis\/blog\/utilizing-the-benzinga-news-archive-to-power-your-llm\/","title":{"rendered":"Utilizing the Benzinga News Archive to Power your LLM"},"content":{"rendered":"\n

Last week Google announced that they would be teaming up with Reddit<\/a> to acquire their historical archive to train their AI models. Human written archives are becoming more and more valuable as AI models and their use cases become more prevalent. In the financial services industry language models can be incredibly useful for algo trading, as well as simply facilitating informed decision-making.<\/p>\n\n\n\n

Now imagine having a tool that not only finds insights into stocks and articles but also breaks down how News impacts the market and its sentiments in a straightforward manner. This is not just an idea; it’s a reality made possible by the vast and valuable data archive that the Benzinga Newsdesk has contributed to over the last 14 years. The Benzinga News S3 bucket houses an extraordinary source of comprehensive data with the potential to power any Large Language Model (LLM), especially one related to finance.<\/p>\n\n\n\n

I’ll guide you through each step, revealing the process of creating your own intelligent assistant. This assistant will excel in reading, summarizing, gaining insights, and analyzing sentiments. Our journey begins by tapping into the data stored in Benzinga’s S3 bucket. We’ll explore how to preprocess this data, a crucial step in extracting meaningful insights. Subsequently, armed with refined information, we’ll delve into the articles, unveiling essential market insights.<\/p>\n\n\n\n

What are S3 buckets and AWS?<\/strong><\/h2>\n\n\n\n

AWS, or Amazon Web Services<\/a>, is like a gigantic toolbox of cloud computing services provided by Amazon. It offers a variety of tools and resources that individuals and businesses can use to run applications, store data, and perform various tasks without needing to set up their own physical servers. Essentially, AWS provides a convenient and flexible way to access computing power and services over the Internet.<\/p>\n\n\n\n

S3 buckets are like containers where you can store and organize your files, such as photos, videos, or documents. Think of them as virtual buckets where you can keep your data safe, and easily access or share it over the internet whenever you need. It’s a simple and efficient way to manage your digital stuff in the cloud.<\/p>\n\n\n\n

Obtaining the Benzinga stock news dataset using AWS S3 buckets <\/strong><\/h2>\n\n\n\n

In this section, we will access the data from the Benzinga’s AWS news database.<\/p>\n\n\n\n

1. We need to install and configure the AWS command line interface first. You can find the download link and instructions here<\/a>. <\/p>\n\n\n\n

2. After installation, you can access AWS from your command line or terminal. Check it through the command:<\/p>\n\n\n

\naws \u2013 version\n<\/pre><\/div>\n\n\n
3. Now, we will configure the command line to access Benzinga containers. Here, you will be asked the username, password, region name, and output format. Use the following command and enter the necessary details for your S3 bucket:<\/p>\n\n\n
\naws configure\n<\/pre><\/div>\n\n\n
4. Once this is done we can access the specific bucket from the AWS account using the command to list all the files or folders there.<\/p>\n\n\n
\naws s3 ls bucket_url\n<\/pre><\/div>\n\n\n
Replace your bucket url from this. Once you cd your self to the directory which you want to download. <\/p>\n\n\n\n
5. Use this command to save it to your desired folder<\/p>\n\n\n
\naws  s3 cp bucket_url path_to_save_folder_ \u2013recursive\n<\/pre><\/div>\n\n\n
The data will be downloaded to the specified folder in the format json.gz<\/p>\n\n\n\n
Converting and Preprocessing Data<\/strong><\/h2>\n\n\n\n
Now armed with our treasure of data, we move on to the crucial step of converting it into a more understandable format \u2013 CSV. A Python script is introduced, demonstrating how to extract and convert JSON data into a CSV file. This step ensures that our model comprehends the data effectively, setting the stage for insightful analysis.<\/p>\n\n\n\n
We will now convert the data to CSV format so that the model understands it. Along with that, we will also preprocess the data:<\/p>\n\n\n
\nimport json\nimport gzip\nimport csv\nimport os\n\n# Function to extract and convert JSON data to CSV\ndef json_to_csv(input_folder, output_csv):\n    with open(output_csv, 'w', newline='', encoding='utf-8') as csv_file:\n        csv_writer = csv.writer(csv_file)\n        # Write the header\n        csv_writer.writerow(["id", "author", "created", "updated", "title", "teaser", "body", "url"])\n         \n        # Loop through each JSON file in the folder\n        for filename in os.listdir(input_folder):\n            if filename.endswith(".json.gz"):\n                file_path = os.path.join(input_folder, filename)\n                with gzip.open(file_path, 'rt', encoding='utf-8') as json_file:\n                    data = json.load(json_file)\n                    print(data)\n                    # Extract relevant fields and write to CSV\n                    csv_writer.writerow([\n                        data.get("id", ""),\n                        data.get("author", ""),\n                        data.get("created", ""),\n                        data.get("updated", ""),\n                        data.get("title", ""),\n                        data.get("teaser", ""),\n                        data.get("body", ""),\n                        data.get("url", "")\n                    ])\n \n \n# Specify the input folder containing JSON files and the output CSV file\ninput_folder = "D:\\\\Scriptonomy Python Developer\\\\ph\\\\01\\\\01"\noutput_csv = "202301---1.csv"\n \n \n# Call the function to convert JSON to CSV\njson_to_csv(input_folder, output_csv)\n<\/pre><\/div>\n\n\n
The provided Python script defines a function, json_to_csv, aimed at converting a collection of gzipped JSON files in a specified input folder into a CSV format. The function iterates through each file, extracts relevant fields such as “id<\/strong>,” “author<\/strong>,” “created<\/strong>,” and others, and then writes this information as rows into a CSV file. <\/p>\n\n\n\n
The script utilizes standard Python libraries such as JSON, GZIP, CSV, and OS for handling JSON data, decompressing files, managing CSV operations, and interacting with the file system. After defining the function, the script specifies the input folder path containing the JSON files and designates an output CSV file. Finally, the script invokes the json_to_csv function, initiating the conversion process and producing a CSV file with the extracted information from the JSON files.<\/p>\n\n\n\n
Making your own article assistant<\/strong><\/h2>\n\n\n\n
Finally, we explore the realm of creating your own article assistant powered by OpenAI’s language model (GPT-3.5-turbo). This assistant aids in sentiment analysis and provides insights into how the information in an article influences market conditions and prices. <\/p>\n\n\n\n
Importing the necessary packages that will help in managing the resources, all these if not present in the environment can be installed with !pip install package_name<\/em>.<\/p>\n\n\n
\nimport openai\nimport pandas as pd\nfrom bs4 import BeautifulSoup\n# openai.api_key = "YOUR_API_KEY"\n\n<\/pre><\/div>\n\n\n