Peaking Duck

PeekingDuck includes some “power” nodes that are capable of processing the contents or outputs of the other nodes and to accumulate information over time. An example is the dabble.statistics node which can accumulate statistical information, such as calculating the cumulative average and maximum of particular objects (like people or cars). This tutorial presents advanced recipes to showcase the power features of PeekingDuck, such as using dabble.statistics for object counting and tracking.

Interfacing with SQL

This tutorial demonstrates how to save data to an SQLite database. We will extend the tutorial for counting hand waves with a new custom output node that writes information into a local SQLite database.

Note

The above tutorial assumes sqlite3 has been installed in your system.
If your system does not have sqlite3, please see the SQLite Home Page for installation instructions.

First, create a new custom output.sqlite node in the custom_project folder:

Terminal Session

[~user/wave_project] > peekingduck create-node
Creating new custom node…
Enter node directory relative to ~user/wave_project [src/custom_nodes]: ⏎
Select node type (input, augment, model, draw, dabble, output): output
Enter node name [my_custom_node]: sqlite

Node directory: ~user/wave_project/src/custom_nodes
Node type: output
Node name: sqlite

Creating the following files:
Config file: ~user/wave_project/src/custom_nodes/configs/output/sqlite.yml
Script file: ~user/wave_project/src/custom_nodes/output/sqlite.py
Proceed? [Y/n]: ⏎
Created node!

The updated folder structure would be:

wave_project/ ⠀
├── pipeline_config.yml
├── src/ ⠀
│   └── custom_nodes/ ⠀
│       ├── configs/ ⠀
│       │   ├── dabble/ ⠀
│       │   │   └── wave.yml
│       │   └── output/ ⠀
│       │       └── sqlite.yml
│       ├── dabble/ ⠀
│       │   └── wave.py
│       └── output/ ⠀
│           └── sqlite.py
└── wave.mp4

Edit the following five files as described below:

  1. src/custom_nodes/configs/output/sqlite.yml:

    1# Mandatory configs
2input: ["hand_direction", "num_waves"]
3output: ["none"]
4
5# No optional configs

    The new output.sqlite custom node will take in the hand direction and the current number of hand waves to save to the external database.

  2. src/custom_nodes/output/sqlite.py:

    Show/Hide Code for sqlite.py

     1"""
 2Custom node to save data to external database.
 3"""
 4
 5from typing import Any, Dict
 6from datetime import datetime
 7from peekingduck.pipeline.nodes.abstract_node import AbstractNode
 8import sqlite3
 9
10DB_FILE = "wave.db"           # name of database file
11
12
13class Node(AbstractNode):
14   """Custom node to save hand direction and current wave count to database.
15
16   Args:
17      config (:obj:`Dict[str, Any]` | :obj:`None`): Node configuration.
18   """
19
20   def __init__(self, config: Dict[str, Any] = None, **kwargs: Any) -> None:
21      super().__init__(config, node_path=__name__, **kwargs)
22
23      self.conn = None
24      try:
25         # try to establish connection to database,
26         # will create DB_FILE if it does not exist
27         self.conn = sqlite3.connect(DB_FILE)
28         self.logger.info(f"Connected to {DB_FILE}")
29         sql = """ CREATE TABLE IF NOT EXISTS wavetable (
30                        datetime text,
31                        hand_direction text,
32                        wave_count integer
33                   ); """
34         cur = self.conn.cursor()
35         cur.execute(sql)
36      except sqlite3.Error as e:
37         self.logger.info(f"SQL Error: {e}")
38
39   def update_db(self, hand_direction: str, num_waves: int) -> None:
40      """Helper function to save current time stamp, hand direction and
41      wave count into DB wavetable.
42      """
43      now = datetime.now()
44      dt_str = f"{now:%Y-%m-%d %H:%M:%S}"
45      sql = """ INSERT INTO wavetable(datetime,hand_direction,wave_count)
46                values (?,?,?) """
47      cur = self.conn.cursor()
48      cur.execute(sql, (dt_str, hand_direction, num_waves))
49      self.conn.commit()
50
51   def run(self, inputs: Dict[str, Any]) -> Dict[str, Any]:  # type: ignore
52      """Node to output hand wave data into sqlite database.
53
54      Args:
55            inputs (dict): Dictionary with keys "hand_direction", "num_waves"
56
57      Returns:
58            outputs (dict): Empty dictionary
59      """
60
61      hand_direction = inputs["hand_direction"]
62      num_waves = inputs["num_waves"]
63      self.update_db(hand_direction, num_waves)
64
65      return {}

    This tutorial uses the sqlite3 package to interface with the database.

    On first run, the node initializer will create the wave.db database file. It will establish a connection to the database and create a table called wavetable if it does not exist. This table is used to store the hand direction and wave count data.

    A helper function update_db is called to update the database. It saves the current date time stamp, hand direction and wave count into the wavetable.

    The node’s run method retrieves the required inputs from the pipeline’s data pool and calls self.update_db to save the data.

  3. src/custom_nodes/configs/dabble/wave.yml:

    1# Dabble node has both input and output
2input: ["img", "bboxes", "bbox_scores", "keypoints", "keypoint_scores"]
3output: ["hand_direction", "num_waves"]
4
5# No optional configs

    To support the output.sqlite custom node’s input requirements, we need to modify the dabble.wave custom node to return the current hand direction hand_direction and the current wave count num_waves.

  4. src/custom_nodes/dabble/wave.py:

    173   ... same as previous ...
174   return {
175      "hand_direction": self.direction if self.direction is not None else "None",
176      "num_waves": self.num_waves,
177   }

    This file is the same as the wave.py in the counting hand waves tutorial, except for the changes in the last few lines as shown above. These changes outputs the hand_direction and num_waves to the pipeline’s data pool for subsequent consumption by the output.sqlite custom node.

  5. pipeline_config.yml:

    11... same as previous ...
12- custom_nodes.output.sqlite

    Likewise, the pipeline is the same as in the previous tutorial, except for line 12 that has been added to call the new custom node.

Run this project with peekingduck run and when completed, a new wave.db sqlite database file would be created in the current folder. Examine the created database as follows:

Terminal Session

[~user/wave_project] > sqlite3
SQLite version 3.37.0 2021-11-27 14:13:22
Enter “.help” for usage hints.
Connected to a transient in-memory database.
Use “.open FILENAME” to reopen on a persistent database.
sqlite> .open wave.db
sqlite> .schema wavetable
CREATE TABLE wavetable (
datetime text,
hand_direction text,
wave_count integer
);
sqlite> select * from wavetable where wave_count > 0 limit 5;
2022-02-15 19:26:16|left|1
2022-02-15 19:26:16|right|1
2022-02-15 19:26:16|left|2
2022-02-15 19:26:16|right|2
2022-02-15 19:26:16|right|2
sqlite> select * from wavetable order by datetime desc limit 5;
2022-02-15 19:26:44|right|72
2022-02-15 19:26:44|right|72
2022-02-15 19:26:44|right|72
2022-02-15 19:26:44|right|72
2022-02-15 19:26:43|right|70

Press CTRL-D to exit from sqlite3.

Counting Cars

This tutorial demonstrates using the dabble.statistics node to count the number of cars traveling across a highway over time and the draw.legend node to display the relevant statistics.

Create a new PeekingDuck project, download the highway cars video and save it into the project folder.

Terminal Session

[~user] > mkdir car_project
[~user] > cd car_project
[~user/car_project] > peekingduck init

The car_project folder structure:

car_project/ ⠀
├── highway_cars.mp4
├── pipeline_config.yml
└── src ⠀
   └── custom_nodes
      └── configs

Edit pipeline_config.yml as follows:

 1nodes:
 2- input.visual:
 3    source: highway_cars.mp4
 4- model.yolo:
 5    detect: ["car"]
 6    score_threshold: 0.3
 7- dabble.bbox_count
 8- dabble.fps
 9- dabble.statistics:
10    identity: count
11- draw.bbox
12- draw.legend:
13    show: ["fps", "count", "cum_max", "cum_min"]
14- output.screen

Run it with peekingduck run and you should see a video of cars travelling across a highway with a legend box on the bottom left showing the realtime count of the number of cars on-screen, the cumulative maximum and minimum number of cars detected since the video started. The sample screenshot below shows:

  • the count that there are currently 3 cars on-screen

  • the cumulative maximum number of cars “seen” previously was 5

  • the cumulative minimum number of cars was 1

PeekingDuck screenshot - counting cars

Counting Cars on a Highway

Note

Royalty free video of cars on highway from: https://www.youtube.com/watch?v=8yP1gjg4b2w

Object Tracking

Object tracking is the application of CV models to automatically detect objects in a video and to assign a unique identity to each of them. These objects can be either living (e.g. person) or non-living (e.g. car). As they move around in the video, these objects are identified based on their assigned identities and tracked according to their movements.

This tutorial demonstrates using dabble.statistics with a custom node to track the number of people walking down a path.

Create a new PeekingDuck project, download the people walking video and save it into the project folder.

Terminal Session

[~user] > mkdir people_walking
[~user] > cd people_walking
[~user/people_walking] > peekingduck init

Create the following pipeline_config.yml:

 1nodes:
 2- input.visual:
 3    source: people_walking.mp4
 4- model.yolo:
 5    detect: ["person"]
 6- dabble.tracking
 7- dabble.statistics:
 8    maximum: obj_attrs["ids"]
 9- dabble.fps
10- draw.bbox
11- draw.tag:
12    show: ["ids"]
13- draw.legend:
14    show: ["fps", "cum_max", "cum_min", "cum_avg"]
15- output.screen

The above pipeline uses the YOLO model to detect people in the video and uses the dabble.tracking node to track the people as they walk. Each person is assigned a tracking ID and dabble.tracking returns a list of tracking IDs. dabble.statistics is used to process these tracking IDs: since each person is assigned a monotonically increasing integer ID, the maximum ID within the list tells us the number of persons tracked so far. draw.tag shows the ID above the tracked person. draw.legend is used to display the various statistics: the FPS, and the cumulative maximum, minimum and average relating to the number of persons tracked.

Do a peekingduck run and you will see the following display:

PeekingDuck screenshot - people walking

People Walking

Note

Royalty free video of people walking from: https://www.youtube.com/watch?v=du74nvmRUzo

Tracking People within a Zone

Suppose we are only interested in people walking down the center of the video (imagine a carpet running down the middle). We can create a custom node to tell PeekingDuck to focus on the middle zone, by filtering away the detected bounding boxes outside the zone.

Start by creating a custom node dabble.filter_bbox:

Terminal Session

[~user/people_walking] > peekingduck create-node
Creating new custom node…
Enter node directory relative to ~user/people_walking [src/custom_nodes]: ⏎
Select node type (input, augment, model, draw, dabble, output): dabble
Enter node name [my_custom_node]: filter_bbox

Node directory: ~user/people_walking/src/custom_nodes
Node type: dabble
Node name: filter_bbox

Creating the following files:
Config file: ~user/people_walking/src/custom_nodes/configs/dabble/filter_bbox.yml
Script file: ~user/people_walking/src/custom_nodes/dabble/filter_bbox.py
Proceed? [Y/n]: ⏎
Created node!

The folder structure looks like this:

people_walking/ ⠀
├── people_walking.mp4
├── pipeline_config.yml
└── src
   └── custom_nodes
      ├── configs
      │   └── dabble
      │       └── filter_bbox.yml
      └── dabble
            └── filter_bbox.py

Change pipeline_config.yml to the following:

 1nodes:
 2- input.visual:
 3    source: people_walking.mp4
 4- model.yolo:
 5    detect: ["person"]
 6- dabble.bbox_to_btm_midpoint
 7- dabble.zone_count:
 8    resolution: [720, 480]
 9    zones: [
10      [[0.35,0], [0.65,0], [0.65,1], [0.35,1]],
11    ]
12- custom_nodes.dabble.filter_bbox:
13    zones: [
14      [[0.35,0], [0.65,0], [0.65,1], [0.35,1]],
15    ]
16- dabble.tracking
17- dabble.statistics:
18    maximum: obj_attrs["ids"]
19- dabble.fps
20- draw.bbox
21- draw.zones
22- draw.tag:
23    show: ["ids"]
24- draw.legend:
25    show: ["fps", "cum_max", "cum_min", "cum_avg", "zone_count"]
26- output.screen

We make use of dabble.zone_count and dabble.bbox_to_btm_midpoint nodes to create a zone in the middle. The zone is defined by a rectangle with the four corners (0.35, 0.0) - (0.65, 0.0) - (0.65, 1.0) - (0.35, 1.0). (For more info, see Zone Counting) This zone is also defined in our custom node dabble.filter_bbox for bounding box filtering. What dabble.filter_bbox will do is to take the list of bboxes as input and output a list of bboxes within the zone, dropping all bboxes outside it. Then, dabble.tracking is used to track the people walking and dabble.statistics is used to determine the number of people walking in the zone, by getting the maximum of the tracked IDs. draw.legend has a new item zone_count which displays the number of people walking in the zone currently.

The filter_bbox.yml and filter_bbox.py files are shown below:

  1. src/custom_nodes/configs/dabble/filter_bbox.yml:

    1# Mandatory configs
2input: ["bboxes"]
3output: ["bboxes"]
4
5zones: [
6   [[0,0], [0,1], [1,1], [1,0]],
7]

    Note

    The zones default value of [[0,0], [0,1], [1,1], [1,0]] will be overridden by those specified in pipeline_config.yml above. See Configuration - Behind The Scenes for more details.

  2. src/custom_nodes/dabble/filter_bbox.py:

    Show/Hide Code for filter_bbox.py

     1"""
 2Custom node to filter bboxes outside a zone
 3"""
 4
 5from typing import Any, Dict
 6import numpy as np
 7from peekingduck.pipeline.nodes.abstract_node import AbstractNode
 8
 9
10class Node(AbstractNode):
11   """Custom node to filter bboxes outside a zone
12
13   Args:
14      config (:obj:`Dict[str, Any]` | :obj:`None`): Node configuration.
15   """
16
17   def __init__(self, config: Dict[str, Any] = None, **kwargs: Any) -> None:
18      super().__init__(config, node_path=__name__, **kwargs)
19
20   def run(self, inputs: Dict[str, Any]) -> Dict[str, Any]:  # type: ignore
21      """Checks bounding box x-coordinates against the zone left and right borders.
22      Retain bounding box if within, otherwise discard it.
23
24      Args:
25            inputs (dict): Dictionary with keys "bboxes"
26
27      Returns:
28            outputs (dict): Dictionary with keys "bboxes".
29      """
30      bboxes = inputs["bboxes"]
31      zones = self.config["zones"]
32      zone = zones[0]         # only work with one zone currently
33      # convert zone with 4 points to a zone bbox with (x1, y1), (x2, y2)
34      x1, y1 = zone[0]
35      x2, y2 = zone[2]
36      zone_bbox = np.asarray([x1, y1, x2, y2])
37
38      retained_bboxes = []
39      for bbox in bboxes:
40         # filter by left and right borders (ignore top and bottom)
41         if bbox[0] > zone_bbox[0] and bbox[2] < zone_bbox[2]:
42            retained_bboxes.append(bbox)
43
44      return {"bboxes": np.asarray(retained_bboxes)}

Do a peekingduck run and you will see the following display:

PeekingDuck screenshot - count people walking in a zone

Count People Walking in a Zone