Project
I subscribe to a data service (IQFeed) that provides live intra-day data. Using a program called Qcollector, individual stock data is collected and stored locally in individual CSV files (one for each stock). These files are updated frequently (at least every 5 minutes).
I need to have three databases built that maintain price data locally for 2000+ stocks on a continuous basis and would like to have a function built enabling MS Excel to read data from the databases very quickly.
Data should be initially collected in three databases reflecting three different time frames – 5 minute data, hourly data, and daily data. If just one database were built, too much intra-day data would need to be stored to capture 2500 periods of end of day data.
The database needs to run in the background collecting current data from the CSV files as well as historic data. It should not require intensive processing capacity as other applications will be running concurrently.
The only data that needs to be collected is closing price – a single data-point for every period for every stock
CSV Files
Each security has its own CSV file. The location of this file, and the name of this file will not change. All CSV files will have a similar file name ([login to view URL]) and will all be stored in the same folder. A sample file is attached ([login to view URL]).
Not all CSV files will have data for exactly the same periods – some might not be as current as others and not all will have the same starting point. It is likely that all will be updated every 5 minutes to reflect new data, but this is not guaranteed.
Data in the database needs to be vetted to ensure that there are no missing periods. Comparability of data across securities is essential. If a data point is missing, the previous data point should be used. If a CSV file is subsequently updated, including previously missing or errant data, the database should be updated to include this corrected data.
Data is time stamped at the end of the period (ie the last price between 9:30 and 9:35am is stamped 9:35am)
Some CSV files will have 24 hours of data per day (currencies – 7pm Sunday through 4pm Friday) while others (stocks) will just have data between 9:30 and 4pm. In the future, foreign securities will be added that have yet different trading periods due to time zone differences.
The following information will be provided in an excel spreadsheet ([login to view URL]) which should be used to determine which securities are to be maintained/updated.
· Security Name
· Security Type
· Security Industry
· Stock Sector
· File Name\Location
· Times for which data is collected (ie 9:30 to 4:00, 24hrs)
Any additions/deletions/changes would be reflected in [login to view URL]
Each CSV file is updated frequently and therefore can't be opened – it must either be queried with-out being opened, or be copied to another location, opened, and subsequently deleted
Database(s)
Initially three databases need to be built to hold Minute, Hourly, and Daily data – 2500 periods of data are needed in each. The data for all three databases will come from the same CSV files. Data from the hourly and daily files can come from the minute database or from the CSV files
Data in the current databases can be limited to 2500 periods, however historic data should be saved in some manner. The amount of data stored in the current database should be flexible (>2500).
Current vs. Future/Incremental timeframes - initially 5 minute data will be used, however this may change to 1 data in the future. Hourly and daily are likely to always remain.
Databases should be updated frequently reflecting the data in the underlying CSV files (update cycle should be variable)
The daily and hourly databases should use the most recent minute datapoint as the last “hour” or “day” (ie use the 2:45pm data point as the end of the day point until 3pm for the hourly database and until 4pm for the daily database).
Additions/Deletions/Changes will need to be made reflecting the fact that certain stocks will be added over time and others deleted as per the [login to view URL] file
The databases should assume 5 business days per week; however, a manual override should be used to allow for holidays when data would not be captured.
The databases need to be transferred easily to other computers such that other computers can read the data locally.
Import into Excel
Build a function that pulls in at least 500 data-points from any of the databases using the following inputs (the import function should look to cells in a spreadsheet to get this information):
· Stock ticker
· Periodicity (5 minute, 10 minute, hour, 2 hour, day, 3 day)
· Starting point (most recent period or historic)
· Number of periods
· Daily period (9:30 – 4pm or 24hrs)
See “[login to view URL]” for a basic file showing inputs/output. Note that date/time data is optional, and that multiple time frames could be drawn from one database.
Already constructed Excel macros will be used to pull in data for multiple stocks and multiple timeframes at once in rapid succession. Hence, any functions built to pull in the data from the databases must be callable by a macro.
Stock name and historic data is only output required (time data is not required if it slows the function down).
Data should always be sorted from most recent at the top to older at the bottom.
Hi This is Piyush Patel
I have 4 yrs of Exp in VB6,VB.net,SQL and MsAccess.
I have developed many applications in VB6.
Please Check My PM
this is only for Demo
Hi,
I would like to do your project. I have 5 year experience in C++.
I have worked on excel when I was in Microsoft. If you don't mind then I would like to do thid project in .NET.
Regards,
Anurag
ICT Makes Sense CC
The More ICT Makes Sense The Better
Good Day. I belive I can assist in the creation of the database as well as the Excel integration. I have been working as a lecturer and consultant for Excel and Databases for the last 3 years. I am also a qualified engineer. Please note this will take time to create correctly and efficiently. Let's give it a go!!!
Respected Sir,
First i want to introduce my company AMAFHH Solutions, a IT company deals in all related feild, we have a different setup for every department.
As we are new to GAF but have strong portfolio.
After seeing your project details, we think that it is not a difficult job for us, you can get result of your choice from us. If you have any querry please dont hesitate to ask, we are just a PM ahead.
Regards,
alilakhany