Matttermost Fails on AWS Beanstalk

For feature requests, please see: http://www.mattermost.org/feature-requests/.

For troubleshooting questions, please post in the following format:

Summary

Matttermost Fails on AWS Beanstalk

Steps to reproduce

I’m using the follow code on AWS BS:
{
“AWSEBDockerrunVersion”: “2”,
“volumes”: [
{
“name”: “app-config”,
“host”: {
“sourcePath”: “/var/app/current/app/mattermost/config”
}
},
{
“name”: “app-data”,
“host”: {
“sourcePath”: “/var/app/current/app/mattermost/data”
}
},
{
“name”: “app-logs”,
“host”: {
“sourcePath”: “/var/app/current/app/mattermost/logs”
}
},
{
“name”: “app-plugins”,
“host”: {
“sourcePath”: “/var/app/current/app/mattermost/plugins”
}
},
{
“name”: “db-data”,
“host”: {
“sourcePath”: “/var/app/current/db/mattermost/var/lib/postgresql/data”
}
},
{
“name”: “web-cert”,
“host”: {
“sourcePath”: “/var/app/current/web/cert”
}
}
],
“containerDefinitions”: [
{
“name”: “db”,
“image”: “mattermost/mattermost-prod-db:latest”,
“memory”: 128,
“essential”: true,
“mountPoints”: [
{
“sourceVolume”: “db-data”,
“containerPath”: “/var/lib/postgresql/data”
}
],
“environment”: [
{
“name”: “POSTGRES_USER”,
“value”: “mmuser”
}, {
“name”: “POSTGRES_PASSWORD”,
“value”: “[snip]”
}, {
“name”: “POSTGRES_DB”,
“value”: “mattermost”
}
]
},
{
“name”: “app”,
“image”: “mattermost/mattermost-prod-app:5.21.0”,
“memory”: 128,
“essential”: true,
“mountPoints”: [
{
“sourceVolume”: “app-config”,
“containerPath”: “/mattermost/config”
},
{
“sourceVolume”: “app-data”,
“containerPath”: “/mattermost/data”
},
{
“sourceVolume”: “app-logs”,
“containerPath”: “/mattermost/logs”
},
{
“sourceVolume”: “app-plugins”,
“containerPath”: “/mattermost/plugins”
}
],
“links”: [
“db”
]
},
{
“name”: “web”,
“image”: “mattermost/mattermost-prod-web:latest”,
“memory”: 128,
“essential”: true,
“mountPoints”: [
{
“sourceVolume”: “web-cert”,
“containerPath”: “/cert”
}
],
“portMappings”: [
{
“hostPort”: 80,
“containerPort”: 80
},
{
“hostPort”: 443,
“containerPort”: 443
}
],
“links”: [
“app”
]
}
]
}

Expected behavior

Successful launch

Observed behavior

[Instance: i-01539c80eaded2057] Command failed on instance. Return code: 1 Output: to start ECS task after retrying 2 times.’ --severity ERROR + exit 1. Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/03start-task.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.

@paulrothrock Would the support team be familiar with this issue?

@ChinyaSuhail, it sounds like the server is failing to start. Can you share the logs from the EB CLI and /var/log/eb-activity.log?

eb-activity.log

@jesse Hello Jesse,
I’m attaching mine as well. I think it’s pretty much the same error with @ChinyaSuhail
Can you give us an insight?

-------------------------------------

/var/log/eb-activity.log

  • RETRY=2
  • sleep 3
  • ‘[’ 2 -gt 1 ‘]’
  • error_exit ‘Failed to start ECS task after retrying 2 times.’ 1
  • error ‘Failed to start ECS task after retrying 2 times.’
  • echo ‘Failed to start ECS task after retrying 2 times.’
    Failed to start ECS task after retrying 2 times.
  • eventHelper.py --msg ‘Failed to start ECS task after retrying 2 times.’ --severity ERROR
  • exit 1 (ElasticBeanstalk::ExternalInvocationError)
    caused by: + . /opt/elasticbeanstalk/hooks/common.sh
  • RETRY=0
  • initctl start eb-ecs EB_EVENT_FILE=/tmp/eventsfile20200602-4066-ncz2qq
    initctl: Job failed to start
  • ‘[’ -f /etc/elasticbeanstalk/.eb-ecs-start-no-retry ‘]’
  • warn ‘Failed to start ECS task, retrying…’
  • echo ‘Failed to start ECS task, retrying…’
    Failed to start ECS task, retrying…
  • eventHelper.py --msg ‘Failed to start ECS task, retrying…’ --severity WARN
  • RETRY=1
  • sleep 3
  • ‘[’ 1 -gt 1 ‘]’
  • initctl start eb-ecs EB_EVENT_FILE=/tmp/eventsfile20200602-4066-ncz2qq
    initctl: Job failed to start
  • ‘[’ -f /etc/elasticbeanstalk/.eb-ecs-start-no-retry ‘]’
  • warn ‘Failed to start ECS task, retrying…’
  • echo ‘Failed to start ECS task, retrying…’
    Failed to start ECS task, retrying…
  • eventHelper.py --msg ‘Failed to start ECS task, retrying…’ --severity WARN
  • RETRY=2
  • sleep 3
  • ‘[’ 2 -gt 1 ‘]’
  • error_exit ‘Failed to start ECS task after retrying 2 times.’ 1
  • error ‘Failed to start ECS task after retrying 2 times.’
  • echo ‘Failed to start ECS task after retrying 2 times.’
    Failed to start ECS task after retrying 2 times.
  • eventHelper.py --msg ‘Failed to start ECS task after retrying 2 times.’ --severity ERROR
  • exit 1 (Executor::NonZeroExitStatus)

[2020-06-02T15:19:23.482Z] INFO [4066] - [Application deployment mattermost-trial-source-1@1/StartupStage1/AppDeployEnactHook/03start-task.sh] : Activity failed.
[2020-06-02T15:19:23.482Z] INFO [4066] - [Application deployment mattermost-trial-source-1@1/StartupStage1/AppDeployEnactHook] : Activity failed.
[2020-06-02T15:19:23.482Z] INFO [4066] - [Application deployment mattermost-trial-source-1@1/StartupStage1] : Activity failed.
[2020-06-02T15:19:23.483Z] INFO [4066] - [Application deployment mattermost-trial-source-1@1] : Completed activity. Result:
Application deployment - Command CMD-Startup failed
[2020-06-02T15:24:33.694Z] INFO [6681] - [CMD-BundleLogs] : Starting activity…
[2020-06-02T15:24:33.694Z] INFO [6681] - [CMD-BundleLogs/AddonsBefore] : Starting activity…
[2020-06-02T15:24:33.694Z] INFO [6681] - [CMD-BundleLogs/AddonsBefore] : Completed activity.
[2020-06-02T15:24:33.694Z] INFO [6681] - [CMD-BundleLogs/BundleLogs] : Starting activity…
[2020-06-02T15:24:33.694Z] INFO [6681] - [CMD-BundleLogs/BundleLogs/BundleLogs] : Starting activity…
[2020-06-02T15:24:33.967Z] INFO [6681] - [CMD-BundleLogs/BundleLogs/BundleLogs] : Completed activity.
[2020-06-02T15:24:33.967Z] INFO [6681] - [CMD-BundleLogs/BundleLogs] : Completed activity. Result:
CMD-BundleLogs - stage 0 - Command CMD-BundleLogs stage 0 completed
[2020-06-02T15:24:33.968Z] INFO [6681] - [CMD-BundleLogs/AddonsAfter] : Starting activity…
[2020-06-02T15:24:33.968Z] INFO [6681] - [CMD-BundleLogs/AddonsAfter] : Completed activity.
[2020-06-02T15:24:33.968Z] INFO [6681] - [CMD-BundleLogs] : Completed activity. Result:
CMD-BundleLogs - Command CMD-BundleLogs succeeded
[2020-06-02T15:27:39.037Z] INFO [6969] - [CMD-TailLogs] : Starting activity…
[2020-06-02T15:27:39.037Z] INFO [6969] - [CMD-TailLogs/AddonsBefore] : Starting activity…
[2020-06-02T15:27:39.037Z] INFO [6969] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2020-06-02T15:27:39.038Z] INFO [6969] - [CMD-TailLogs/TailLogsDockerPS] : Starting activity…
[2020-06-02T15:27:39.038Z] INFO [6969] - [CMD-TailLogs/TailLogsDockerPS/DockerPS] : Starting activity…
[2020-06-02T15:27:39.105Z] INFO [6969] - [CMD-TailLogs/TailLogsDockerPS/DockerPS] : Completed activity.
[2020-06-02T15:27:39.105Z] INFO [6969] - [CMD-TailLogs/TailLogsDockerPS] : Completed activity. Result:
CMD-TailLogs - stage 0 - Command CMD-TailLogs stage 0 completed
[2020-06-02T15:27:39.105Z] INFO [6969] - [CMD-TailLogs/TailLogs] : Starting activity…
[2020-06-02T15:27:39.105Z] INFO [6969] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity…
[2020-06-02T15:27:39.351Z] INFO [6969] - [CMD-TailLogs/TailLogs/TailLogs] : Completed activity.
[2020-06-02T15:27:39.351Z] INFO [6969] - [CMD-TailLogs/TailLogs] : Completed activity. Result:
CMD-TailLogs - stage 1 - Command CMD-TailLogs stage 1 completed
[2020-06-02T15:27:39.351Z] INFO [6969] - [CMD-TailLogs/AddonsAfter] : Starting activity…
[2020-06-02T15:27:39.352Z] INFO [6969] - [CMD-TailLogs/AddonsAfter] : Completed activity.
[2020-06-02T15:27:39.352Z] INFO [6969] - [CMD-TailLogs] : Completed activity. Result:
CMD-TailLogs - Command CMD-TailLogs succeeded
[2020-06-02T15:37:34.956Z] INFO [7769] - [CMD-TailLogs] : Starting activity…
[2020-06-02T15:37:34.956Z] INFO [7769] - [CMD-TailLogs/AddonsBefore] : Starting activity…
[2020-06-02T15:37:34.956Z] INFO [7769] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2020-06-02T15:37:34.956Z] INFO [7769] - [CMD-TailLogs/TailLogsDockerPS] : Starting activity…
[2020-06-02T15:37:34.956Z] INFO [7769] - [CMD-TailLogs/TailLogsDockerPS/DockerPS] : Starting activity…
[2020-06-02T15:37:35.023Z] INFO [7769] - [CMD-TailLogs/TailLogsDockerPS/DockerPS] : Completed activity.
[2020-06-02T15:37:35.023Z] INFO [7769] - [CMD-TailLogs/TailLogsDockerPS] : Completed activity. Result:
CMD-TailLogs - stage 0 - Command CMD-TailLogs stage 0 completed
[2020-06-02T15:37:35.024Z] INFO [7769] - [CMD-TailLogs/TailLogs] : Starting activity…
[2020-06-02T15:37:35.024Z] INFO [7769] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity…
[2020-06-02T15:37:35.251Z] INFO [7769] - [CMD-TailLogs/TailLogs/TailLogs] : Completed activity.
[2020-06-02T15:37:35.251Z] INFO [7769] - [CMD-TailLogs/TailLogs] : Completed activity. Result:
CMD-TailLogs - stage 1 - Command CMD-TailLogs stage 1 completed
[2020-06-02T15:37:35.251Z] INFO [7769] - [CMD-TailLogs/AddonsAfter] : Starting activity…
[2020-06-02T15:37:35.251Z] INFO [7769] - [CMD-TailLogs/AddonsAfter] : Completed activity.
[2020-06-02T15:37:35.251Z] INFO [7769] - [CMD-TailLogs] : Completed activity. Result:
CMD-TailLogs - Command CMD-TailLogs succeeded
[2020-06-03T02:09:49.388Z] INFO [16781] - [CMD-TailLogs] : Starting activity…
[2020-06-03T02:09:49.388Z] INFO [16781] - [CMD-TailLogs/AddonsBefore] : Starting activity…
[2020-06-03T02:09:49.388Z] INFO [16781] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2020-06-03T02:09:49.388Z] INFO [16781] - [CMD-TailLogs/TailLogsDockerPS] : Starting activity…
[2020-06-03T02:09:49.388Z] INFO [16781] - [CMD-TailLogs/TailLogsDockerPS/DockerPS] : Starting activity…
[2020-06-03T02:09:49.454Z] INFO [16781] - [CMD-TailLogs/TailLogsDockerPS/DockerPS] : Completed activity.
[2020-06-03T02:09:49.455Z] INFO [16781] - [CMD-TailLogs/TailLogsDockerPS] : Completed activity. Result:
CMD-TailLogs - stage 0 - Command CMD-TailLogs stage 0 completed
[2020-06-03T02:09:49.455Z] INFO [16781] - [CMD-TailLogs/TailLogs] : Starting activity…
[2020-06-03T02:09:49.455Z] INFO [16781] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity…

I found some related issue at HERE
If I understand correctly, Mattermost officieal AWS Beanstalk Installation Guide is not reproducible…at this moment…right?

Thanks @ChinyaSuhaila and @johnnykoo84 – I had hoped the logs would shed some light on why Mattermost is failing to start, but it seems like it’s just the equivalent of systemd logs without the necessary context.

I’m not intimately familiar with AWS Beanstalk, but can you find and share the Mattermost server logs emitted from this context?

Thanks for the response @jesse I’ll be glad to provide as much as i can, but at this moment i’m not entirely sure where I could possibly find the mattermost server logs… perhaps I should ssh into EC2 and manually export the log? @chinyaSuhail do you have any clue on this?