I am trying to build my lambda function using SAM. My lambda function depends on a custom python function which I plan to build as an AWS Lambda layer. My custom python function has a transitive dependency on a python package publicly available on PyPI, which I specified in the requirements.txt file for the layer.

Here's the folder structure for my lambda function:

my-lambda-func
|-- events
    |-- event.json
|-- my-func
    |-- my_function.py
    |-- __init__.py
    |-- requirements.txt
|-- my-layer
    |-- my_layer.py
    |-- requirements.txt
|-- template.yaml 
|-- buildspec.yml

Here's my template.yaml file:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-test

  Sample SAM Template for sam-test

Globals:
  Function:
    Timeout: 300

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: my-func/
      Handler: app.lambda_handler
      Layers:
        - !Ref MyLayer
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api
          Properties:
            Path: /hello
            Method: get

  MyLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
        LayerName: MyLayer
        Description: My custom Lambda Layer
        ContentUri: ./my-layer/
        CompatibleRuntimes:
            - python3.8
        LicenseInfo: MIT
        RetentionPolicy: Retain
    Metadata:
        BuildMethod: python3.8

My buildspec file:

version: 0.2

phases:
  build:
    commands:
      - sam package --template-file template.yaml --s3-bucket my-bucket --output-template-file packaged-template.yml

artifacts:
  files:
    - packaged-template.yml

when I run a sam build command locally, I see the 2 resources getting created correctly including pulling the transitive dependencies for the layer but when I setup CodePipeline with a build phase to build the command from builspec file and a deploy phase using CloudFormation, the layer isn't built as I expect it to. When I download the layer from the console, it doesn't show any transitive dependencies.

Has anyone done anything similar to this. Can somebody help with what I am doing wrong? Thanks!