2326982 - Background Jobs in LMS Knowledge Support and Tips

 

Click to go back to the main page

BACKGROUND JOBS IN LMS KNOWLEDGE SESSION FOR CUSTOMERS, PARTNERS AND SAP PRODUCT SUPPORT

1. Background Jobs Overview

2. Important Configurations

2.1. Important LMS_ADMIN Configurations

2.2. REPORT_SYSTEM Configurations

3. Scheduled Job Workflow

4. Scheduled Job in Database

4.1. PA_SCHEDULED_JOB table Information

4.2. PA_SCHEDULED_JOB Statuses

4.3. PA_SCHEDULED_JOB Columns

5. Timeouts

6. Frequently Used APMs and Notifications

6.1. Learning Plan APM

6.2. Learning Expiration APM

6.3. Evaluation Synchronization APM

7. KBA related to Background Jobs

 

  1. Background Jobs Overview

Background jobs are used in the application where a process is too large to be ran on the dedicated server sections (nodes) that the application runs on.  The job instead is ran a specific node dedicated to only running the jobs (background jobs node) so that the applications performance (on the applications dedicated nodes) is not affected.

Nodes for SuccessFactors Learning

• Application (usually 3 or more) – All application processes run here and the node name contains ‘u’, like XXXXu02.
• Reporting – Reports ran as Online are ran here and the node name contains ‘r’, like XXXXr01.
• Background Jobs – Reports that are scheduled, application processes, and Automatic Processes are ran here.  These nodes contain the value ‘b’ in their name
• Connector Jobs– This node is a background jobs node, but only for connectors and the name contains ‘c’, like XXXXc01.

Types of Background Jobs:

• Reports – reports can be ran online (on the reporting nodes) or scheduled if the results returned will be too large (scheduled on the background jobs node)
• Application processes – Large processes in the application such as mass assignments, removals, and any other changes in the LMS that may take place that exceed the applications threshold in the LMS_ADMIN configuration will be scheduled as a job(discussed in detail in later slides).
• Automatic Processes (APM) – System processes that are scheduled and automatic once enabled.  They will also mass assign, remove, and more importantly send large amounts of notifications each time they are ran based on their schedule.
• Connector Processes – Connectors are also scheduled as Background Jobs.

2. Important Configurations

Most of the Background jobs configurations can be found in System Admin > Configuration > System Configuration in the below sections.

LMS_ADMIN
Configurations for system processes, notifications and APM background jobs

REPORT_SYSTEM
Report related configurations that apply when a report is scheduled to be ran as a background job (not online, ie on the reporting nodes)

 

2.1. Important LMS_ADMIN Configurations

There is a configuration for the times of day and for each day of the week that a job can be ran.  Below is the configuration for Monday. Based on this the jobs can be ran on Monday (enabled) and within all 24 hours of the day and on any date, month, or year where a Monday falls on.
allowableBackgroundJobTimeframes[Monday].enabled=true
allowableBackgroundJobTimeframes[Monday].startMinute=*
allowableBackgroundJobTimeframes[Monday].startHour=0
allowableBackgroundJobTimeframes[Monday].startDate=*
allowableBackgroundJobTimeframes[Monday].startMonth=*
allowableBackgroundJobTimeframes[Monday].startYear=*
allowableBackgroundJobTimeframes[Monday].startDayofWeek=mon
allowableBackgroundJobTimeframes[Monday].endMinute=*
allowableBackgroundJobTimeframes[Monday].endHour=24
allowableBackgroundJobTimeframes[Monday].endDate=*
allowableBackgroundJobTimeframes[Monday].endMonth=*
allowableBackgroundJobTimeframes[Monday].endYear=*
allowableBackgroundJobTimeframes[Monday].endDayofWeek=mon

Background jobs are scheduled when the below threshold is met:
 # If the number of records that a operation operates on exceed this count the operation may have to be scheduled as a background job to run later instead of running online
 # Assignment Profile jobs also affected by the Assignment Profile Job specific settings, below.
onlineOperationMaxRecordsCount=5

As currently set, if a process requires to update more than 5 entities in the database (6 assignments, 6 removals, or 6 learnings recorded, etc.), the system will then schedule the transaction as a background job so that the application nodes performance is not affected.

# Specifying the batch size greater than 0 (zero) enables the processing of the records in batches.
# This gives the ability to the user to commit the transaction after each batch has been processed. Default: 1000
numberOfRecordsToBeProcessedInABatch=1000

This setting if greater than 0 will process the amount of changes in batches.  For example, if an item is assigned to 5900 users, there will be 5 batches of 1000, and the final batch for the remaining  900, all in the same job, just broken down into chunks/batches.

# enableMakeupFiring allows the last late firing to occur. For example, suppose a job is scheduled to fire once a day, the scheduler is down for a day, and the
# late time window is small enough such that all these firings would be skipped when the engine begins running again. If makeup firings are not enabled, then
# these late jobs will not fire. On the other hand, if makeup firings are enabled, then the last late job will fire.
enableMakeupFiring=true

This setting sets the number of emails that will be sent at the same time.  If 20,000 emails are to be sent from PA_MAIL (mail queue in the database), they will be sent in 2 separate groups of 10,000 based on the above configuration.  The frequency of each batch that will be sent is determined in minutes in the Email Frequency setting in System Admin > Configuration > Global Variables > Mail tab.

# Specifies the maximum number of e-mail messages that will be purged from mail archive at any one time. Default: 10000
emailArchivePurgeBatchCount=1000

If email archiving is on in System Admin > Configuration > Global Variables > Mail, once the “Purge Emails older than” threshold is met this configuration will remove the mail in batches of 1000 until all emails are removed that are older than the number of months in the Purge Emails older than: setting.  Archived emails are found in the PA_MAIL_ARCHIVE table if the Enable Email Archiving is enabled in System Admin > Configuration > Global Variables > Mail.

 

2.2. REPORT_SYSTEM Configurations

# Timeout period for running reports online (not as a scheduled background job) in seconds.  If a report takes longer than this
# to generate, it is automatically timed out.
onlineTimeoutPeriod=300

If a report job is not scheduled (Online), this setting is in effect to time the job out if it exceeds the number of seconds set (5 minutes in the above).

# Timeout period for running reports as a scheduled background job in seconds.  If a report takes longer than this
# to generate, it is automatically timed out.
backgroundTimeoutPeriod=3000

If a report job is scheduled, this setting is in effect to time the job out if it exceeds the number of seconds set (50 minutes in the above).

These settings are to ensure that jobs do not bog down the servers performance based on where the report jobs are ran – reporting node (online) or background jobs node (scheduled).

#List the individual report names separately that can only be scheduled to run in the background.
#In order for the Admins to determine the correct report name to enter they will need to go through
#the Report Export process, in the first step of that process the wizard will display the report name
#that they are trying to export.
scheduleOnlyReports.UserCurriculumStatusCSV=true

This setting is where you can list individual reports that can only be scheduled.  Most reports listed here will return a large amount of data, so the option to run Online is removed entirely.

3. Scheduled Job Workflow

4. Scheduled Job in Database

NOTE: this section it is more for Customer Support Troubleshooting

4.1. PA_SCHEDULED_JOB table Information

All scheduled jobs have a job ID and can be found in this table.

Each run of the current job ID will be recorded in PA_SCHEDULED_JOB_RESULT table and identified by the JOB_RESULT_ID.  The PA_SCHEDULED_JOB_RESULT table will also data  for older job IDs and is basically a job history table.

As previously mentioned, Background jobs are scheduled when the below threshold is met:

  # If the number of records that an operation operates on exceed this count the operation may have to be scheduled as a background job to run later instead of running online
  # Assignment Profile jobs also affected by the Assignment Profile Job specific settings, below.
onlineOperationMaxRecordsCount=*

 

4.2. PA_SCHEDULED_JOB Statuses

JOB_STATUS column

Scheduled – job has a scheduled time and yet to start
Pending – Job has started and is initializing/preparing to begin
Executing – Job is executing/running
Succeeded – Job finished successfully with no errors or problems
Aborted – The job (reports) has not completed within the timeout period set in REPORT_SYSTEM
Failed - The job failed with errors.
Canceled – The job was canceled during the executing phase by an admin (OWNER).

 

4.3. PA_SCHEDULED_JOB Columns

OWNER – Admin/user who last scheduled the job. The Admin ID must exist in the LMS/DB.

There will be errors (the errors will state that the Admin ID is missing) if the following admin IDs do not exist in the DB, but are listed as the OWNER in the PA_SCHEDULED_JOB table:

CONNECTOR (Applied as LST_UPD_USR in tables the connector run updates)
AUTO-SCHEDULER (Owner of the Learning Plan APM in version 6.4(b12xx) to now)
LAST_JOB_RESULT_ID is the last JOB_RESULT_ID, or the last run of the job found in PA_SCHEDULED_JOB.  A job can only have one job_id, but will have a job_result id in the PA_SCHEDULED_JOB_RESULT table for every run while the job stays as scheduled.

In PA_SCHEDULED_JOB_RESULT, the PROBLEMS column will show any errors the job may have encountered.  Again, each time the job runs, this table will contain the info for each based on the JOB_RESULT_ID.

 

 Some jobs like the Assignment Profile Synchronization will Lock if other jobs affect it.  The lock if in place can be found in PA_SCHEDULED_JOB under the LOCK_ID column and will need to be removed by Operations.

 

Two columns that can be used to find a specific job by name (not job_id) are the INSTANCE_ID and DESCRIPTION.  For example, if I am looking for the Learning Expiration APM, I could use the below:  INSTANCE_ ID LIKE ‘%Expiration%’   or DESCRIPTION LIKE ‘%Expiration%’

 

 

The JOB_EXECUTION_SERVICE_ID will tell you the exact server node that the job was ran on in PA_SCHEDULED_JOB_RESULT for each execution of the job (job_id).

The TYPE column will tell you what kind of job it is.  TYPES are APM, ImmediateReport, RecurringReport, AdHoc (AdHoc job types usually update records, are jobs created for APMs, or send emails), Report
ENABLED (Y or N) will show if the job is enabled or not

LAST_UPDATE_USER will show the last admin ID to make a change to the job and LAST_UPDATE_TIMESTAMP will show when that change occurred.

5. Timeouts

Does this happen with all jobs, some, or a specific job?
These questions will help determine the scope of the problem and whether the issue lies with an entity that all processes use (DB package, trigger, etc), or a specific entity only used by the job in question.

Does it involve a Standard or custom report?
This helps to determine whether the problem lies with the background jobs node or report library that all reports use (all reports affected), or if it’s a specific custom report where you would then focus on that report.  There maybe an issue with the custom query or the contents of the library that the report is using.  For reporting, this question helps to narrow down where you should be focusing.

What is the full error from the reporting node, or background jobs node if it was scheduled?
The error usually helps shed light on the problem, either in the first few lines or the last (caused by).  Its important to pull logs and check the DB as soon as possible if an error is present.

How much data is involved?  Does the job/report run fine on less criteria/data returned?
Timeouts usually occur when a report is returning a large set of data.  Getting the scope of how much data is to be returned based on the tables the report pulls from will help determine if it’s a performance issue (too much data), or an issue with the report itself (minimal data returned and should not be encountering a time out).

Is the STATUS in PA_SCHEDULED_JOB of the report as 'Initializing', or did the report begin to generate data?
The Status will help determine what phase of the report is causing the issue.  Timeouts usually occur in the beginning phase when the report is getting ready to run and basically taking a quick glimpse of the amount of data its about to process.

‘No progress was reported for *** seconds’ usually points to the JobProgressTimeout found on the server taking effect.  These jobs are often huge and get stuck at the ‘Initializing’ STATUS in PA_SCHEDULED_JOB, and usually require Operations

The other Timeout setting available is for reports in the REPORT_SYSTEM configuration mentioned before.

 

6. Frequently Used APMs and Notifications

The three most widely used Automatic Processes are the Assignment Profile Synchronization, Learning Plan, and Learning Expiration, with the AP Sync being the most widely used.

• Assignment Profile Sync (AP Sync) – is the job that syncs all Assignment Profiles that are propagated.  Un-propagated APs are ignored.  In PA_ASSGN_PRFL table, the AP_ID will have a job_id if propagated.  Once the job completes, it sends the status of the job to the email address specified on the scheduling page (System Admin>Automatic Processes>Assignment Profile Sync) to admin email listed on the APMs scheduling page using
• BackgroundJobNotification – sends the results (start time,  end time, status,   error, job, type etc.)
  ApmSynchronizeStudentChangesAssignmentProfile – email address specified on the scheduling page showing all the AP’s that were synced (not all changes, but ones that were touched).

**It’s important to note that the AP Sync job will lock and quit running if the User Connector is running at the same since the connector will be updating users records that the AP sync job uses to assign to users.

 

6.1. Learning Plan APM

Testing the Learning Plan APM

The Learning Plan APM gathers all assignments and removals since the last run in the PA_LEARNING_PLAN tables.  The records are not put in until the APM runs.  The next time it runs, it compares what’s there vs what’s currently in PA_STUD_CPNT and PA_STUD_QUAL_CPNT.

Any changes made since the last APM run in the STUD_CPNT tables are validated on the next APM run and if found, the ‘items modified’ email is sent.  Once removed from the Learning Plan (and the PA_STUD_CPNT tables), the records are also removed from the PA_LEARNING_PLAN tables since the APM no longer needs to notify on them unless they are added to the users plan again, and then added back to PA_LEARNING_PLAN upon the APM running again.

Regardless of whether its enabled or disabled, the table still gathers the changes (until the items are removed for the users learning plan), and upon the next run, it sends the notifications out for all changes since the last run, so it’s best to keep enabled if desired to avoid confusion.  One issue usually seen with disabling is if the APM was disabled at the first of the month, and then enabled at the end, all the assignments, changes, and removals within the month will still be sent.

Once enabled and ran, it populates the changes of learning plan assignments and removals since the last run based on the PA_LEARNING_PLAN tables and pushes to the PA_MAIL table.  The SendMailApm job (internal heavyweight APM that is scheduled when ‘Enable Email’ is checked) runs based of the frequency in the in the Configuration> Global Variables > mail settings (email frequency) and sends out mail based on a background batch configuration.

The PA_LEARNING_PLAN table also stores information on what was sent, but only once the mail enters PA_MAIL to then go out via the SMTP (mail) server.

**Sent emails can also be Archived if it’s enabled in Global Variables.  The archive is only in the database and in the PA_MAIL_ARCHIVE table.

The main issues seen with these APMs are with the Item and usually it points to them in the caused by part of the error.  Sometimes the AUTO_SCHEDULER admin that usually owns the jobs was deleted and needs to be reincarnated (created again).  Jobs like the Learning Plan APM and Learning Expiration will experience errors like “OWNER: AUTO-SCHEDULER does not exist” if the admin ID is missing from the LMS.

• ApmStudentLearningPlanNotification is the template that the student (user) receives
• ApmSupervisorLearningPlanNotification is the template that the supervisor (user) receives

 

6.2. Learning Expiration APM

The expiration APM checks a users’ REQUIRED BY (REQ_DTE in DB in PA_STUD_CPNT) date and once the date is within the threshold set on the Item > Notifications section (let’s say 30 days) it sends a notification using the below template, and then flags the user in the DB with a reminder date (PA_STUD_CPNT – RMNDR_DTE column) x amount of days ahead where x is the reminder date set.
 
If a customer changes the runtime of the APM, the reminder dates (RMNDR_DTE) are not automatically updated and runs can potentially skip notifications.  The Scheduled time is meant to not to be changed frequently.
 
Once the Required date has passed, only the Overdue (if enabled) notifications are sent based on the reminder period (the threshold has no value after the item(s) are overdue).

Its also important to note that EACH item has its own Threshold and Reminder periods and they can be individually changed in the items’ Notifications section.

• ApmTrainingExpiration – notification sent when threshold is met and then x number of days (reminder period setting) afterword's
• ApmTrainingOverdue – Sent when the item is past due based on the required date (b1408)

 

6.3. Evaluation Synchronization APM

If you have an optional or required survey (item evaluation or follow-up (assigned if enabled in the issued completion status)) assigned on your learning plan (gets assigned upon completion of the item which survey is attached to), this APM will generate a notification to the user, and a notification to the supervisor if it’s a follow up and the option for BOTH user and supervisor is checked.

It will send the notification for the ‘Allow’ period that is set if it’s a follow –up (allow being user has X amount of days to complete, and assign being x number of days after item is complete that survey should be assigned).  Testing purposes would require assign to be 0 so that the survey is immediately assigned.
 
This APM uses the ItemEvaluationAssignmentNotification – Sent to users with Item Evaluations are pending on their learning plans.  Notification is sent with each run and there is no threshold/reminder period.

7. KBA related to Background Jobs

2232736 - How to change Background Jobs schedule time range

2075340 - Resync Learning Timing out

2312363 - LMS: Learning notifications being sent to admin email configured on background job

2279925 - Background Report Jobs Fail: Please contact your administrator for more details on the error

2211479 - How to enable the APM 'User Requested Offering Notification‘

2216839 - Parent key not found error when running Deny Expired Approval Processes APM

2216933 - What does notification Interval mean in the recommendations newsletter settings APM?

2219839 - VLS schedule Offering Automatic Completion using VLS attendance Processing APM – LMS

2250169 - Learning Expiration APM doesn't Trigger Email as per the settings set at the Threshold of an Item - LRN-11134

2269911 - APM Deny Expired Approval Process is failing

2277719 - Supervisor Receives Email for Expiration, even after setting unchecked at APM level

2279410 - Purge Expired Catalog items APM

2315568 - Threshold and reminder period for APM learning expiration

2323243 - APM generating http deeplink instead of https

2184360  - VLS Attendance Processing APM error:java.lang.NullPointerException: while trying to get the length of an array loaded from local variable 'vleEnrollSeats'- LMS

More and more KBA's are created everyday by our Product Support Engineers, to review more KBA's related to Background Jobs in LMS clikc on the link.

To review more KBA's related to Automatic Process (APM) click on the link.

Image/data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental.

 

SAP SuccessFactors Learning Management System

APM, Background Jobs LMS, LMS Background Jobs, Automatic Process, LMS_ADMIN, REPORT_SYSTEM, PA_SCHEDULED_JOB, Learning Plan APM, Learning Expiration APM, Evaluation Synchronization APM , KBA , LOD-SF-LMS , Learning Management System , LOD-SF-LMS-ADM , System Admin, Global Variables, References , LOD-SF-LMS-BCJ , Assign (bulk), Add/ Delete Sub/ Cur, Revise, Import , LOD-SF-LMS-NOT , Notifications , How To

SAP SuccessFactors Learning all versions