This project was originally a project to get the cost of groceries by scraping sites and then rendering it on a lovely UI.
Prototype demo and explanation on youtube
Since then, I've removed my AWS infrastructure, but I've kept the project alive. Right now I use it to render a CSV file to the terminal so that I can see which store is overall better. An example is this google sheets page here
Docker ,Angular, Nginx, FastAPI, Python, AWS, Postgres, SSL, Certbot.
This is the sales pitch about why this is cool, where the complexity is in the project.
- Good example of multi-stage docker images.
- Good use of Nginx as a reverse proxy.
- Example of using FastAPI and Docker to perform web-scraping.
- Generating and using SSL certificates to make applications more secure.
- How networking works in the full-stack of an application lifecycle.
- Using ECR to manage docker images to build locally with a lot more ram rather than on a poor t2.micro.
- Using Postgres as a database solution with RDS.
Python FAST API wrapped in a docker container. API endpoints will reach out to retailers, scrape, and insert data to the DB.
Angular v11.1.4 behind an Nginx reverse proxy in a docker container that talks to the backend.
- EC2 t2.micro instance to host backend and frontend.
- Security groups configured (443/8000) to allow traffic in and out of the VPC.
- RDS instance created to store data.
- Route53 for DNS
- Certbot / LetsEnrypt used for SSL.
CREATE TABLE product (
id serial PRIMARY KEY,
catagory VARCHAR NOT NULL,
description VARCHAR NOT NULL,
retailer VARCHAR NOT NULL,
price FLOAT NOT NULL,
last_updated TIMESTAMP NOT NULL ,
brand VARCHAR NOT NULL,
sku VARCHAR NOT NULL,
url VARCHAR UNIQUE NOT NULL,
other VARCHAR
);
CREATE TABLE historical_prices (
id serial PRIMARY KEY,
url VARCHAR NOT NULL,
price FLOAT NOT NULL,
last_time TIMESTAMP NOT NULL,
CONSTRAINT productIDCon FOREIGN KEY ( url ) REFERENCES public.product ( url )
);
ALTER TABLE historical_prices ADD UNIQUE(url);
- A feature I would like to Add is have a historical_price where we just note a price and a product ID.
- To generate the SSL certificates we used Certbot :
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo certbot certonly --manual --preferred-challenges=dns --email onuallainc@gmail.com --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d *.taifuwiddies.net
- Storing historical prices and updating current products.
- Fixed scrolling on many results returned.
- Overall website theme to be less dark.
- Added last updated field to the UI.
- URL colour changed.
- Fixed the CSS.
- Added rendering of a CSV to see prices.
- Support CSV output.
- Support Dunnes stores.
- Enable ranking of companies for best products.
- Enable Grouping of products by store.
- Support price per unit for all stores.
- MultiProcessing supported to make things faster.
- Hyperlink URLs.
- Establish base list of requirements for a week.
- Weekly price seeding.
- What are each company promoting specials on.
As it stands right now I have it configured to render data to terminal.
note sometimes on macs docker networking may not be able to connect to the internet and this will do nothing.
docker build -t grocer_back -f backend.Dockerfile .
docker run grocer_back
./bin/build.sh