AWS Data Query Advice

$20-50 USD

Closed

Posted about 6 years ago

$20-50 USD

Paid on delivery

I'm completely new to AWS but have a project that I need to get underway. I have some c#/.net experience so I need some advice at this stage from someone that has experience with data querying/analytics and natural language query. In a nutshell, I am looking for the best way to populate a AWS datastore for the purpose of adding natural language query (using Alexa) and reporting and analytics/predictive analysis. I have 3 data entities that I am currently logging to S3 buckets in json format. The files are arrays of json records (10 - 1000 records per file) with a well known stable schema (serialized c# classes) that is not likely to change, (or very little). The raw data is coming from some EC2 VMs with a custom syslog server, where logs are filtered and transformed into structured c# classes and periodically flushed to S3 buckets in files of 10 -10,000 objects. Initially, I thought I might need to use AWS Glue to ETL this into some sort of indexed datastore but perhaps this is not the best approach? I read that parquet format is used by big data analytics so one option is that I could probably save the files into parquet format using parquet.Net. My thinking is that we might as well save the data in the most useful format for the platform and possibly skip ETL jobs? But this is where I have no experience and need some advice. What would be the most cost effective way to save data to s3 in the first place? If I save in parquet format, might this be ready for querying directly without ETL? What is the best way I can save the data in terms of buckets, folders, files and the naming conventions for each. (Note: we will keep ~1 month of data in the datastore and then use lifecycle policy to archive to Glacier.) Or is there merit in storing the data into an aws database (which?) as we are starting with fully structured data? Can you help with your real world experience with this and perhaps our next steps of setting up querying and natural language? So this component (mini-project) is just for expert advice.

Amazon Web Services

Data Warehousing

Data Analytics

Project ID: 24246209

About the project

3 proposals

Remote project

Active 6 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

3 freelancers are bidding on average $35 USD for this job

@ranumehta2017

***AWS EXPERT*** Hi, Hope you are doing great !! I have major work experience in Server Administration and Project Management. AWS Services : EC2, S3, RDS, CloudFront and many more.I provide all kinds of solution related to Network administration for system admin, Linux, Unix, Apache, servers, lambda EC2, Open SSL. I am grateful for your time and consideration,and I look forward to speaking with you further about this position.I am willing to work to work for 40 hrs per week for your project if you hire me once Warm Regards, Ranu

$50 USD in 1 day

4.7

(28 reviews)

4.6

@Aloksingh1166

AWS expert Developer ready to ROCK! Dear Concern, I am writing to apply for the position, as advertised on freelancing website. My nearly 9 years of experience in creating custom automation robots similarly to uipath to achieve Robotic Process automation includes responsibilities such as Project Management, Software Development, Coding Requirement and Gathering. I am currently working as the Team Lead. My key skills include C#,ASp.net,Rpa,java,python,Docker,sql,mysql,html,php,javascript,Apis,jquery,uipath,opencv,automation,node.js,powershell,shellscript Have good Experience in automating web ui elements using selenium web driver. This role requires me to plan project activities by scoping, estimation, tracking, risk management, delivery management & post implementation support; document, track, communicate bugs, enhancements, analysis and unresolved problems; liable for managing full end-to-end process for the user acceptance testing phases of the project, executing testing & working within deadlines. I bring to your company my years of relevant experience and my drive for results and positive outcomes. I am prepared for the next challenge in my career and look forward to hearing from you. Thank you for your time and consideration.

$35 USD in 7 days