This repo hosts code for the Data Commons website.

Data Commons is an open knowledge graph that provides a unified view across multiple public data sets and statistics. We've bootstrapped the graph with lots of data from US Census, CDC, NOAA, etc., and through collaborations with the New York Botanical Garden, Opportunity Insights, and more. However, Data Commons is meant to be for community, by the community. We are excited to work with you to make public data accessible to everyone.

To see the extent of data we have today, browse the graph using our browser.

In https://github.com/datacommonsorg/website, click on "Fork" button to fork the repo.

Clone your forked repo to your desktop.

Add datacommonsorg/website repo as a remote:

git remote add dc https://github.com/datacommonsorg/website.git

Every time when you want to send a Pull Request, do the following steps:

git checkout master
git pull dc master
git checkout -b new_branch_name
# Make some code change
git add .
# Run tests
./run_test.sh -a
git commit -m "commit message"
# If tests pass
git push -u origin new_branch_name

Then in your forked repo, you can send a Pull Request. If this is your first time contributing to a Google Open Source project, you may need to follow the steps in CONTRIBUTING.md.

Wait for approval of the Pull Request and merge the change.

This repo contains DataCommons mixer repo as a submodule.

When mixer upstream is updated, and you want to track the change:

git submodule foreach git pull origin master

Contact Data Commons team to get permission for access of secret manager.

Contact Data Commons team to get permission for GCP application credential. Then run the following command once

gcloud config set project datcom-browser-staging && \
gcloud auth application-default login

This will generate a json file in your local machine, record the path and set the environment variable in your ~/.bash_profile or ~/.bashrc file

export GOOGLE_APPLICATION_CREDENTIALS=<JSON_CREDENTIAL_PATH>

Before running client-side tests and packages, install nodejs

Before running the tests, install the browser and webdriver. Here we recommend you use Google Chrome browser and ChromeDriver.

• Chrome browser can be downloaded here.

• ChromeDriver can be downloaded here, or you can download it using package manager directly:

  npm install chromedriver

If using Linux system, you can run the following commands to download Chrome browser and ChromeDriver, this will also include the path setup:

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb; sudo apt-get -fy install
CHROMEDRIVERV=$(curl https://chromedriver.storage.googleapis.com/LATEST_RELEASE)
wget https://chromedriver.storage.googleapis.com/${CHROMEDRIVERV}/chromedriver_linux64.zip
unset CHROMEDRIVERV
unzip chromedriver_linux64.zip
sudo mv chromedriver /usr/bin/chromedriver
sudo chown root:root /usr/bin/chromedriver
sudo chmod +x /usr/bin/chromedriver

Note: Make sure that your ChromeDriver version is compatible with your local Google Chrome version. You can change view the lastet ChromeDriver version here.

./run_test.sh -a

cd static
npm test testfilename -- -u

./run_npm.sh

This will watch static files change and re-build on code edit.

Start the flask webserver locally at localhost:8080

./run_flask.sh

Install "Air" for Go server auto reload

go get -u github.com/cosmtrek/air

Start the Go webserver locally at localhost:7070

air

Another way to start the server locally is to use Docker.

Make sure you have installed Docker Desktop.

Build Docker image

docker build -t datacommonsorg-website .

Run it locally

docker run \
-p 8080:8080 \
-e GOOGLE_APPLICATION_CREDENTIALS=<JSON_CREDENTIAL_PATH> \
-v $GOOGLE_APPLICATION_CREDENTIALS:<JSON_CREDENTIAL_PATH>:ro \
datacommonsorg-website

This file is stored in GCS bucket: datcom-browser-prod.appspot.com (for prod) and datcom-browser-staging.appspot.com (for local and staging). To update this file, please contact Data Commons team.

Check Redis instance information.

gcloud config set core/project datcom-browser-staging(datcom-browser-prod)
gcloud redis instances describe redis-instance --region=us-west2

Apache 2.0

For general questions or issues, please open an issue on our issues page. For all other questions, please send an email to support@datacommons.org.

Note - This is not an officially supported Google product.