In this blog we will cover how to build an ML powered bird classification solution using a DeepLens, the Caltech CUB-200-2011 dataset and the ResNet 18 neural network architecture. The solution includes automatic image uploads to a Hugo Easy Gallery on S3. The project is still a bit in the scrappy MVP state but read on for the high level build flow. Check out the Bird Brain Gallery on my Hugo site.

Prerequisites

This post assumes S3 and Hugo are used for site hosting and DeepLens is used for image capture and inference. However, the solution approach could be used with other devices such as Raspi with additional building.

Build Steps

There are 4 main stages for this project:

  1. Configure DeepLens device
  2. Deploy the Bird Classification project template
  3. Modify the project code
  4. Configure Hugo and S3

Configure DeepLens device

Once your DeepLens is registered and online in your AWS account, mount the unit with a clean viewing angle to the feeder. The DeepLens unit is not designed for outdoor use so this is just temporary to get things working. The ultimate plan is to install the DeepLens indoors with a window mounted bird feeder or possibly a vented enclosure…

 

Deploy Bird Classification project template

In the AWS DeepLens console, create a new project using the Bird Classification template and deeploy it to your DeepLens. Once deployed, use the output stream to confirm the image capture is suitable, i.e. the birds are inside bounding box.

 

Modify the project code

In the Bird Classification project, click the Lambda funtion in the and replace the code in greengrassHelloWorld.py with this modified code for S3 upload functionality. Add your desired S3 bucket (where Hugo site is hosted) name for the bucket_name variable. The prob_thresh variable sets the confidence threshold percentage to capture an image. The bird_hist_ variable determines how many images to retain in the target folder. To keep things simple, the images rotate through fixed names _bird-#.jpg_. This helps with a static site like Hugo where the gallery index will automatically reflect the new images as they are uploaded.

Create an IAM Policy with s3:PutObject permissions to the desired Amazon S3 Bucket, adding that policy to AWSDeepLensGreengrassGroupRole.

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"PutObject",
      "Effect":"Allow",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource":["arn:aws:s3:::YOUR-BUCKET/*"]
    }
  ]
}

Configure Hugo and S3

Create the directory for your gallery, i.e. /static/image/birdgallery and load seed images based on the bird_hist variable in your greengrass function. The stub images are needed for the gallery index to be built and will be overwritten by the uploaded images from DeepLens. The images are statically named bird#.jpg and can be seeded in the gallery by unzipping these images and using the following AWS CLI command from the directory with the unzipped images:

aws s3 sync ./ s3://your_hugo_bucket/img/birdgallery/ --content-type "image/jpeg" --cache-control "max-age=1200"

The cache control is set to 20 minutes to refresh the images for viewers as they rotate. When updating your Hugo content in the future, be careful to preserve the gallery index and not to overwrite the gallery images.

Configure your Hugo install with Easy Gallery and add the shortcode to your relative path for your S3 site.

{{< gallery dir="/img/birdgallery" >}} {{<> load-photoswipe >}}

If everything is working, the DeepLens should begin uploading images to your Hugo gallery when the configured probability threshold is reached. In addition to uploading the image to S3, the DeepLens also sends the prediction as JSON to an IoT topic with the following format:

'$aws/things/{}/infer'.format(os.environ['AWS_IOT_THING_NAME'])

You can view the messages with the MQTT client in the console or create rules to send the data to targets like S3, SNS, Elasticsearch etc. Here is a sample JSON message.

{
  "format": "json",
  "payload": {
    "Downy_Woodpecker": 0.5231890625
  },
  "qos": 0,
  "timestamp": 1614004813209,
  "topic": "$aws/things/deeplens_YYYYYYYYY_ZZZZZZZZ.infer"
}

The model accuracy / mAP isnt perfect but it has successfully predicted several species.

 

There are a few improvements I would like to make:

  • A different bird feeder setup with better view and less movement
  • Aggregate the JSON prediction data and use Chart.js for visualizations
  • Improve model performance through additional training data using captured images

Summary

In this blog we covered how to build an ML powered bird classification solution that automatically uploads bird images to S3 for viewing in Hugo Easy Gallery. The Lambda code and artifacts for this solution are available on GitHub.

Happy back yard ML birding!

 

- Cameron