Node.js Flashcards
What does Mosh hope we become after this course?
(Hint: Superstar Node developer)
A superstar node developer
Why?
Most comprehensive, most up to date node course
Shows all modern ways to build applications in node
What is node?
an opensource, cross platform
runtime environment for running Javascript code outside browser
Used to build services that power client apps (Web app, Mobile App)
What is node ideal for building?
Highly scaleable, data intensive, real time backed services that power applications
What is so special about node?
(Hint: How is it different from ASP.net, Ruby on Rails, Django)
(faster developement)
Great for prototyping and agile development
(better performance)
Build superfast and highly scaleable backends
Ex. Used by Paypal, Netflix, Uber
(more developers)
anyone with Javascript skills can work as a full stack developer
(cleaner, more consistent source code)
same tools, conventions and best practice (Javascript full stack)
(largest open source ecosystem)
free open source libraries to use
What big tech companies use Node?
(Hint: Uber, Paypal, Netflix, Ebay, NASA)
“We have seen with our proof of concept a tremendous amount of improvement on the response times: for our current APIs we have a steady response of 2 seconds and the Node.js API that we hosted was responding within less than 100 milliseconds. That’s a very promising statistic for us.”
Netflix
node.js app reduced startup time, allowed more movie streaming
Paypal
node.js app outperformed java app on page load times
mobile software stack build solely in Node.js
Yahoo
Node.js has become one of their top technologies
Uber
Node.js into full production allowing quick iterations and zero risks for matching drivers with riders
Groupon
50% reduce in page loading times
GoDaddy
10x less servers thanks to Node.js
Ebay
faster iterations, 175 million users
Walmart
allows fixing bugs much faster
NASA
creating microservices architecture with separate APIs, moving data on astronaut space suits allowing faster queries
What did Paypal find when they rebuild their Java application in node.js?
(Hint: faster, less code, fewer files, faster response time)
An excellent choice for building highly scaleable applications
Where was Javascript run time environment originally?
(Hint: JS engine in browsers)
JS engine converts Javascript code into machine code
Each browser has it’s own JS engine
Ex.
Chrome - v8 engine
Firefox - SpiderMonkey
Internet Explorer - Chakra
Therefore,
sometimes Javascript code behaves differently in different browsers
Why did Ryan Daal embed Chrome’s V8 engine into a C++ program?
V8 engine is fastest Javascript engine
allows us to execute Javascript outside browser
Node.js will convert Javascript code into machine code
gives us additional objects (file system, network, server requests)
What is node.js NOT?
(Hint: Language, Framework)
Not a programming language (C#, Python, Java)
Not a Framework (ASP.NET, Django)
It’s a runtime environment for executing Javascript code!
Provides v8 engine to translate Javascript to machine code
Offers additional objects for working with servers, networks, etc.
What feature of node.js allows it to build highly scaleable, super fast applications?
(Hint: asychronous framework)
Asynchronous
non blocking nature of node.js
ideal for intensive Input/Output apps
ideal for apps that require heavy disk or network access
Why?
Node is continuously monitoring event cue for finished tasks
While event queue is empty, serves additional clients
can serve more clients with less hardware
What should node.js NOT be used for?
(Hint: CPU intensive applications)
Do not use node.js for CPU-intensive apps
video encoding or image manipulation service
Why?
Node is single threaded, while CPU runs, clients must wait
lots of work required by CPU, low requests to file system or network
What should node.js be used for?
ONLY USE for data intensive, real time applications
DO NOT USE for CPU intensive apps
Why?
Single thread get’s caught up with CPU calculations
What core modules in node.js are explored?
(Hint: OS, fs, events, http)
also learn how to create our own modules
What is the global scope object in browsers?
Window
all variables, objects and functions declared globally are accessible in this object
What is the global object in node?
(Hint: global)
variables and functions defined are not automatically added to global scope
in browsers, it is!
Why?
Scoped to their file, because of Node.js module system
Why is it a problem if variables and functions are added to the global scope?
(Hint: maintainable and reliable applications)
Javascript code often split into multiple files
if function is added to global scope
when we define function in another file
definition will override this defintion
Soln?
to build maintainable and reliable applications
create modularity
avoid adding functions and variables to the global object
How can we build maintainable and reliable applications in node?
(Hint: Modules)
Using modularity
modules or files provide abstraction and private members
create small building blocks (modules) where we define variables and functions (encapsulated in module)
variables and functions are scoped to that module (private)
not accessible outside the module unless we export them
Every file in node is a module
Why?
So that variables and functions don’t override eachother
If we want to use a variable or function outside a module, what must we do?
(Hint: export and make public)
Modules provide abstraction
making variables and functions private (not accessible globally)
to use a member defined in a module, must export it
What does every node application have?
(Hint: one module or file)
At least one module or file
Ex. app.js
How do we create and load a module?
Like classes in Java
module.exports
require( )
As a best practice, store module object in a constant
Why?
So we don’t accidentally override it
What is JSHint?
(Hint: Tool for catching compile time errors)
Can scan Javascript for compile time errors
When do we export an object vs. a function in a module?
(Hint: can export object or single function)
Exporting an object is useful if we have multiple functions or properties
(export an object)
module.exports.member = member
(export a single function)
module.exports = member
How does node create a module?
Doesn’t execute code directly
Wraps it inside a function (module wrapper function)
What are some of the useful modules built into node?
Streams
OS - work wit operating system
HTTP - create web servers
file system - work with files
process - get info on current process
query string - usefull for http services
What is the path module?
(Hint: Paths are used in URLs to specify file locations)
Built in node module
better than working with strings
useful methods for working with paths
What is the OS module?
Built in node module
gives us information about current operating system (OS)
os.freemem() - gives free memory
os.totalmem() - gives total memory
os.uptime() - up time of computer
os.userinfo() - current user information
What is the file system module?
(Hint: working with file systems)
Node built in module
always prefer to use asynch methods
What is an event?
(Hint: core functionality in node)
A signal that something has happened in our application
node’s core functionality is built around handling events
(EventEmitter)
Ex.
HTTP listening to port, client request incoming, raises event
application reads request, returns right response
What is an event argument?
(Hint: additional tasks when handling request)
adding additional arguments when raising an event
can pass data about event that just occured
listener can recieve data (event arg) and call url, log, etc.
event arg best practice - encapsulate in obj
How do we work with EventEmitter in the real world?
(Hint: encapsulate in a class)
Not directly
create a class with all capabilities of EventEmitter
use this class in application
What is the HTTP module used for?
(Hint: creating networking applications)
One of the powerful building blocks of node
ex. create webserver that listens for HTTP request
can create backend for React or Mobile
However…
In real world, we don’t use HTTP module
Why?
As we add more routes, code get’s complex, added linearly in callback function
Use express instead (built on top of HTTP module)
What is node package manager (NPM)?
(Hint: largest software registry in the world)
A command line tool and free open-source registery
for third party libraries (re-usable code)
any functionality for an app, most likely
a free open-source library (package) on NPM registery
1 million packages
11 million developers rely on re-usable, open source tooling
Share?
Can create and add to NPM
How do we install a package using npm?
(Hint: > npm i package)
How does node resolve modules?
(Hint: require( ‘./module’) )
- First, checks to see if the module is in the node core
- Next, checks to see if the module is in a folder/file in the project
- Finally, checks to see if the module is in the node_modules folder
How do we load / use a third party module?
(Hint: underscore.js module)
How do we install a third party module?
(Hint: npm mongoose)
Package.JSON updates dependencies
What should we exclude from our source control repository?
(Hint: node_modules)
exclude node_modules
Why?
It could be hundreds of megabytes in size (large)
everytime someone checks out code, have to download
Soln?
package.json stores all node module dependencies
can use npm to instal dependencies locally!
How can we exclude node_modules from git?
(Hint: don’t commit node_modules to source repo)
Create a .gitignore file
What is semantic versioning (SemVer)?
Hint: Major.Minor.Patch
Ex. mongoose 4.13.6
major version
add new feature that could break existing apps dependent on old version
minor version
adding new features, API not broken, increase minor
patch release
bug fixed, increase patch version
What does the ^ character mean?
(Hint: Semantic versioning)
It tells node…
download the current module_package as long as the major version is the same (no breaking changes will occur)
if newer minor or patch version is available, will download
Ex. Built application using mongoose ^4.13.16
six months later, someone checks out code from repository
downloads dependencies…
newer mongoose version available 4.14.0
(as long as same major version)
new minor and patch will be downloaded/installed
What does the ~ character mean?
(Hint: same major, same minor, updated patch)
Tells node to download the dependent module with…
same major version
same minor version
updated (newest) patch version
What should we do if we want someone to get exact version of a dependent module?
(Hint: remove ~ or ^)
Sometimes in real world, new patch versions break previous versions
To ensure exact module versions are downloaded…
remove ~ or ^ from dependencies in package.json
How can we see a list of all the packages (modules) installed in modules folder?
Hint: npm list
Terminal…
npm list (shows all dependencies)
To show only your app…
npm list –depth=0
How can we view information on a package (module)?
Hint: npm view package
returns package.json for module
can also go to package registry (npm registry)
check dependecies
look for current versions
How do we view only the dependencies of a package (module)?
Hint: npm view dependencies
Ex. npm view mongoose depencies
How do we downgrade or upgrade a node package?
Hint: installing a specific version
Get versions:
> npm view packagename versions
Downgrade/upgrade:
> npm i packagename@major.minor.patch
How can we view and update outdated node packages?
Hint: npm outdated
Terminal:
>npm outdated
>npm update
**only minor and patch versions updated
How to get major releases?
>download npm-check-updates
How do we update a node package to it’s current version?
Hint: use npm-check-updates command line tool
> npm-check-outdated (ncu)
> ncu - u (update)
> npm i (install)
What is the difference between application dependencies and development dependencies (DevDependencies)?
(Hint: App vs. Development)
App dependencies
required for our app to function properly
Dev Dependencies
used during development process (testing, analysis, bundling code)
**should not go in production environment where APP deployed
>npm i jshint –save-dev
package.json will segregate devDependencies and appDependencies
How do we uninstall a package (module)?
Hint: npm uninstall packageName
> npm uninstall packageName
or
> npm un (uninstall)
package.json updated and node_modules updated
How do we add a package to node package manager?
> npm login
> npm publish
Success!
Now, use in another project
> npm i package
*under dependencies in package.json
How can we update a package we have published on the npm registry?
specify version update (major, minor or patch)
npm will update version
Ex. npm version minor
v1.1.0
What are the key lessons about node package manager?
What problem does Express solve?
Hint: building lightweight, fast applications
We don’t want to hardcode if statements
What is Representational State Transfer (REST)?
Hint: Restful services (APIs)
Introduced by PhD student as his thesis
a convention for building HTTP services:
(CRUD) operations:
Create, Read, Update, Delete
*expose resources on various endpoints (customers list)
What is the client-server architecture?
client (front end)
sends HTTP requests to server (using HTTP protocol)
to save data, store data, access data
What are the standard HTTP methods?
Hint: GET, POST, PUT, DELETE
Every HTTP request has a verb that determines intention
Application exposes resource
using meaningful address
get resource, add to resource, delete from resource, update resource
GET
getting data
Ex. GET /api/customers to get list of all customers
response will be an array of customer objects
POST
creating data
POST /api/customers post new customer to collection
include customer object in body of
PUT
updating data
PUT /api/customers/1
include complete customer object in body with updated properties
server updates this customer object in customer resource
DELETE
deleting data
DELETE /api/customers/1
What is Express?
Hint: popular framework for managing HTTP routes
Express is most popular framework for managing routes
Fast, lightweight and perfectly documented
Why?
As we define more routes,
we have to add more conditional statements
A framework like Express
allows us to add additional routes while keeping our code clean and maintainable
How is using Express different than using node’s HTTP module?
We don’t have if blocks
define new routes using app.get( )
can move routes to different files
Express gives our app a skeleton (structure)
Ex.
can move all routes for courses to courses.js
What is an environment variable?
(Hint: proper way to set port in node apps)
a variable that is part of environment process runs
value is set outside of our application
Ex. Port environment variable
Process is a global object with env
Why?
In real world, port is dynamically assigned based on hosting environment
Cannot hard code it and expect port will work
What are route parameters?
essential (required) values passed with a route
Ex. Get all posts from a given year / month
Server - /api/posts/:year/:month
Client - /api/posts/2020/08
What are query string parameters?
Additional data sent to our server in HTTP request
Used to provide additional data (optional data) to backend service
Stored in a query object
Ex. /api/posts/2020/02?sortBy=name
How do we handle a GET request using Express?
How do we use POST method in Express?
app. use(express.json())
app. post
As a security best practice what should you never, ever do when working with HTTP services?
Hint: NEVER Trust what the client sends
Always validate client input
NEVER EVER trust what client sends
Soln?
NPM Joy
What is Joi?
(Hint: Client Validation)
An NPM package that simplifies validation of client POST requests
How?
provides a schema (how many chars, input allowed, etc.) for POST requests
Why?
Don’t want to validate client input at the beginning of POST request
When we send an invalid request to our server using joi, what do we get?
an error object
too complex to send to client
instead, access error message in the object
.error.details[0].message
How do we handle HTTP put requests using Express?
Look up data
If not exists, return 404
validate client req
if invalid, return 400
update data
return updated data to client
What are the advanced topics about Express covered?
Middleware
function that takes request obj
returns response to client or passes control to another middlewear function
core concept in Express
Configuration
storing configuration settings and loading for different environments
Debugging
debug package in node.js
Templating Engines
returning HTML markup to client (vs. JSON)
Pug, Mustache, EJS
Generating dynamic HTML to return to client
What is a middleware function?
An essential concept in Express
Express includes pre-built middleware
Can create custom middleware for request processing pipeline
(every request goes through request processing pipline)
Ex. logging, etc.
MIddleware function takes a request obj
AND
terminates req/response cycle by returning response to client
OR
passes to another middleware function
Ex. app.get(‘/’, (req, res) => {})
How are our middleware functions called?
In sequence
Ex. First logging, then authentication
What should we do with our custom middleware functions?
Define custom middleware in a separate module
Import (require() ) and install (app.use())
Why?
clean code practice
What are third party Middleware?
Express.js pre-built middleware
But
every third party middleware used will decrease performance (request processing time) of application!
So use wisely and sparingly
Best Practice Middleware?
Helmet - helps secure application
Morgan - log http requests
How can we enable special middleware in specific environments and not others (production, staging, development)?
NODE_ENV=Production
What is the config module?
Hint: Express third party middleware
Used for setting configuration settings - production, development, testing
can store config settings for appliction
DO NOT:
application secrets
password of database or mail server
that password/secret is available
WHY?
Check into repository, everyone can see
SOLN?
Store in env variable
How do we store our application’s secrets?
(database password, mail server password)
Hint: environment variables
export app_password=1234
custom-environment-variable.json (file)
What is the debug module in node?
Used for debugging
can set various environments for debugging
when in that environment, only debug messages from that environment shown
Ex. export DEBUG=app:*
(get all debug environment variables)
What is a templating engine?
Servers often return JSON objects
sometimes need to return HTML markup to client
Templating engines help us generate dynamic HTML / return to client
Popular:
PUG
Mustache
EJS
How should we structure our applications using Express?
Don’t write all code in index.js
First, create a folder called routes, put route logic for each API endpoint in a separate file (module) in this folder.
Import each module in index.js
Ex. courses.js and home.js which contain all the logic for the routes to courses API and home API
Second, create a folder called middleware and move logic for custom middleware to this module
How do we integrate Express with a database?
(Hint: SQL, Mongo, etc. )
Head over to Express.js and view their docs on database integration
Mosh covers mongo and mongoose later in the course in depth!
How is authentication covered in Express?
It’s NOT a part of Express
authenticating users to access API endpoints is covered in another module
What does the config package give us?
What is asynchronous code?
Asynch does not mean concurrent (multi-threaded)
single thread executes first line,
schedules next task for future
returns to execute next line of code
returns to complete task
Ex. Single waiter in a restaurant (single thread)
takes order, doesn’t wait in kitchen for cheff
only one waiter (not multithreaded)
What are the three patterns for asychronous code?
Callbacks
a function that is called when the result of an asynchronous operation is ready.
Promises
async and wait
syntactical sugar over promises
What is a callback function?
a function that is called when the result of an asynchronous operation is ready.
When the result is ready, call the result back using a function
Why?
Cannot return a value from an async function (it’s undefined until task complete)
Must use a callback to call result when it’s ready
Ex. Call the user object back when result is ready
What is callback hell aka christmas tree implementation?
Nested callback functions
Solution?
flatten structure of the code
replace anonymous function with a name function
second arg to callback is anonymous function
How can we solve callback hell?
Hint: Flatten the structure
Nested structure exists because second arg to callback is an anonymous function
Soln?
flatten structure of the code
replace anonymous function with a name function
Best?
Use promises
What is a promise?
extremely powerful when dealing with asychronous Javascript code
an object that holds the eventual result of an asynch operation
What three states can a promise object be in?
(Hint: Promise is an object that holds the eventual result of an async operation)
Pending
initially when we create a promise object it’s in pending state, then it kicks of an asych operation
Fulfilled
asynch operation is completed with a value
Rejected
if something went wrong during execution of asynchronous task, result will be an error
How do we write a promise?
Promise is an object that holds eventual result of an async operation
Initially, it’s in the pending state
if the result is completed, it moves from pending to resolved (fulfilled)
resolve - sends result to consumer
if there is an error, it moves from pending to the rejected
reject - returns an error to consumer
How do we replace callbacks with promises?
Modify callback functions to return promises
How do we consume promises?
Promises expose .then ( ) method
can chain to create complex asynch operations
How do we create already resolved or already rejected promises?
(for unit testing)
Promise.reject()
Promise.resolve()
How do we kick off multiple async operations at the same time?
Hint: not concurrency, single threaded
Single thread kicks off multiple promises at same time
not waiting for result of first asyc operation to complete
kicking off next asyc operation right away
Waiting for ALL promises to complete
returns an array
If any of our async operation fails, final promise is considered rejected
How do we kick off multiple promises and continue when the first completes?
Promise.race()
As soon as one promise in array is completed, promise at end of race is considered resolved
Promise resolves as soon as first promise resolves
Final Result is value of first fulfilled promise
What is the await approach to writing asyncronous javascript?
await the result of a promise, store it and pass it to another async function
Why?
It’s cleaner and easier to understand than callbacks and promises
Remember:
Whenever you use an await inside a function
You must decorate the function with the async keyword
What are Async and Awaits?
Syntactical sugar over Promises
Allow us to write async code that looks sync
Built over promises
JS engine converts our code into Promises
Code looks synchronous but it’s executed asynchronously
the JS engine executes this code asynchronously
await releases thread to do other work
Code still runs async
looks synchronous
errors are caught using try/catch block
What database are we going to use in this course?
Hint: MongoDb
MongoDb
noSQL database
Why?
A very popular database management system
Often used in Apps built in Node and Express
Expert?
MongoDb requires it’s own course
learn enough to get the job done here
What is MongoDb?
A very popular database management system
Often used in Apps built in Node and Express
How different?
No rows, columns
no schema or database design
simply store JSON objects in MongoDb
Query MongoDb get JSON objects
Simply, return JSOB objects to clients
How do we setup MongoDb?
https://stackoverflow.com/questions/57856809/installing-mongodb-with-homebrew
> brew tap mongodb/brew
>brew install mongodb-community
>brew services start mongodb-community
Create directory:
> sudo mkdir -p /data/db
store mongodb data
> sudo chown -R id -un /data/db
intialize mongo
> mongod
deamon listens in background for port connections
What is MongoDB compass?
(Hint: connect / view mongoDB data)
A client application to connect to a mongodb server
can look at database
can work with our data
How do we connect to MongoDB?
Using Mongoose
What are the equivalents to tables and rows in MongoDb?
Collections and Documents
Collections
Document
each document is a container of key value pairs
Mongoose schema defines shape of each document
What is a schema in mongoose?
Hint: Document Validation Abstraction over MongoDb
Helps us define the properties (shape) of each document
Mongoose.Schema( { object } )
object passed defines key value pairs for each document
What are the schema types in Mongoose?
String
Number
Date
Buffer
Boolean
ObjectID
Array
How do we save a document in MongoDb?
Create a document
Store in MongoDb
How do we query documents using Mongoose?
Can filter, limit number of documents, etc.
How do we write comparision queries using Mongoose?
.find ( { key: { } } )
pass an object for the value in a key, value pair
eq (equal)
ne (not equal)
gt (greater than)
gte(greater than or equal to)
lt (less than)
lte (less than or equal to)
in
nin (not in)
How do we use comparision operators to query documents using Mongoose?
.or ( )
pass an array or filter objects
How do we use regular expressions when querying a document using mongoose?
.find ( )
pass an object with (key, /pattern/)
^ starts with (start of string)
$ ends with (end of string)
/i make not case sensitive
.* zero or more characters present
How do we implement pagination using mongoose?
.skip ( (pageNumber - 1) * pageSize )
.limit ( pageSize )
How do we write code to query MongoDb to display…
backend courses only
published status
sorted by name
returned as only the author/course name?
.find ( )
.sort ( )
.select ( )
How do we update a document in MongoDB?
Hint: Query first vs. Update first
Query First -
We want to validate permissions
Update First -
Just want to update directly
Ex. Update Likes in DB without returning Post object
How do we delete a document in MongoDB?
How do we write validation in our mongoose Schema?
wrap in an object
name: {type: String, required: true}
validation kicks in at the time we try to save a document
Remember…
Validation implemented on Schema properties are only meaningful in Mongoose
MongoDB doesn’t care about validation logic implemented in Mongoose
- *It doesn’t implement validation at Data Base level like SQL**
- *So…**
Validation code is only meaningful to Mongoose
Mongoose runs validation logic when we try to save a document,
if document is not valid,
it will not save it to the database
How do we conditionally validate a field in our mongoose schema?
Ex. Price is only required if course is published
add required validator in an object
return a boolean from required function
Cannot use => and .this
Why?
Because => .this references enclosing function
function ( ) { .this} references current object
What are the mongoose built in validators?
Number
min, max
String
minlength, maxlength, match, enum
How do we create custom validation in mongoose?
How do we write code for an asynchronous validator in mongoose?
Use promises!
callbacks depreciated
isAsync: true
callback( )
What are SchemaType objects?
Used to assert validation on schema fields Mongoose
custom getter/setter available
What is a best practice for structuring our applications using mongoose?
Hint: extract models (single responsibility API vs. models)
Create a models folder
extract logic for creating model into separate file
include schema validation logic in separate file
How can we work with related objects in mongoDB?
Tradeoff between query peformance (embedded documents) and data consistency (references)
References aka normalization
Consistent data but requires additional query for getting the reference
Embedded Documents (denormalization)
Single query to get both documents
queries must run as fast as possible? use embedded docs
What is the hybrid approach to working with document relationships in mongoDB?
Embed only half (parts) of another document (not full document)
How do we reference an object in mongoose?
{type: mongoose.Schema. Types.ObjectId,
ref: Author}
References aka normalization
Consistent data but requires additional query for getting the reference
How do we query a reference object in mongoose?
.populate (‘property’)
query performance is reduced by having a second query to get reference object
How do we write code for embedding a document inside a document (denormalization)?
set property to a schema
subdocuments are same as reguar documents
but can only be saved in context of parent
How can we embed an array of subdocuments in a document?
Hint: embedding docs to model relationships between documents
How do we create the movies API (model and router) using Express, Mongoose and Node?
(Hint: embedded subdocument)
Why don’t we need to worry when embedding partial members of a subdocument?
don’t reuse the schema created in original model files
documentID will be there
can query database for full info if needed in the future
Why?
subdocuent could have 50 properties, we don’t want to embed them all
How do we create the rentals API using mongoose, express and node?
How does MongoDb implement transactions?
Using Two Phase Commit
Fawn 3rd party library does it for us!
Is every document object ID unique?
Virtually, yes
if more than 16 million documents are being updated at same time (in same process, on same machine),
_id: will become same, rare chance
_id: generated by driver, not MongoDb
Why?
Don’t have to wait for MongoDB to create a unique identify
MongoDB driver can create a unique identifier
mongoose talks to mongoDB driver to generate a unique ID
What makes MongoDB so highly scaleable?
generate object ID in memory, not MongoDB
ID generation by MongoDB driver, indpendent of MongoDB
Don’t have to wait for MongoDB to create a unique identify
MongoDB driver can create a unique identifier, pass to mongoDB
mongoose talks to mongoDB driver to generate a unique ID without MongoDB
How do we validate object id’s in Joy?
There is an npm plugin for validating objectIDs
>npm i joi-ObjectId
What are the key factors to deciding whether or not to embedd a subdocument or reference a subdocument?
Data consistency - references (one copy of data), slower reads
Performance - embedded subdocuments (multiple copies of documents, data inconsistency), faster reads
What is authentication and authorization?
Nearly, all applications require some authentication / authorization
Authentication:
The process of identifying if user is who they claim to be
aka
logging in - send username, password to server
Authorization:
determining if user has right permission to perform operation
only logged in users can perform operations to modify data
not logged in, can only read from API endpoints
only admins can delete data (permissions)
How?
Add two new API endpoints
Register: POST /api/users
Login: POST /api/logins (creating new resource, login)
What is Lodash?
Extremely powerful
optimized version of underscore
utility functions for working with strings, arrays, etc.
How do we enforce strong passwords using Joi in our Node applications?
How do we hide the user’s password in the server’s 202 response?
use loadash
_.pick(object, [paths])
How do we enforce password security?
Hint: Joi-password-complexity
What can hackers do to passwords that are hashed without a salt?
cannot decript applications hash
can compile a list of popular passwords, hash them
look at our database, find what our passwords are
a salt is an arbitrary string placed before our password
Why?
Hash password will be different each time based on the salt
How do we hash user’s passwords using a salt?
npm i bcrypt
What is a JSON web token?
On client store long string (JSON token)
send to server
A long string that identifies a user, stored on client
functions like a drivers license or password
when client logins in, server generates JSON web token
next time, client must send JSON web token for future API calls
on client, store JSON web token to send for future API calls
Stored?
Web application - local storage on every browser (React, Angular)
Mobile APP - similar options
What is the anatomy of a JSON web token?
Stored on client, sent to server with each client request
(Header)
alg - algorithm used to encode JSON token
(Payload)
basic public properties about user, sent to server with client request
can extract name, user_id, permissions, etc. without additional query to database from JSON web token
(Digital Signature)
digital signature created based on content of JSON web token + private key (only on server)
Hacker cannot modify payload without re-creating the digital signature using the private key (only on secured server)
How do we generate a JSON web token?
const jwt = .require(‘jsonwebtoken’)
jwt.sign( { payload } , ‘privateKey’)
How do we store private keys / secrets?
Hint:
Never store secrets in source code
$ export vidly_jwtPrivateKey=mySecureKey
npm config package
config package - store config settings in JSON files / environment variables
Steps:
install config (npm i config)
create config file in project
create default.json
create custom-environment-variables.json
replace secret in app with call to config.get ( )
secret stored in environment variable - export vidly_jwtPrivateKey=mySecureKey
(default.json)
to store template for config settings
(custom-environment-variables.json)
specify mapping between app settings and environmental variables
Ex. Store jwtPrivate key using config
How do we store our application’s private key in an environment variable?
Hint: app private key used in JSON web token digital signature
How do we enable continuous authentication?
Hint: user registers and stays logged in
Send JSON webtoken in header as response to client
store JSON web token on client
with each request, send JSON web token
How do we implement authorization?
Hint: giving users permissions for modifying data
Create a middleware function
re-use in route handlers that require permissions
router (route, *middleware, callback)
*executed before callback
How do we return the current user to the client?
How do we implement roles?
middleware function
What are the key points of authentication and authorization?
What is a JWT?
How do we authenticate a user?
add a method to the userSchema prototype
What are important JWT and authorization notes?
How do we handle errors properly?
Send a friendly error to client
log the exception on the server
How?
using promises - catch() rejected promise
using await/async - wrap in try/catch blocks
Why?
In real world, issues happen
logging the issues to see which happen frequently is important
Ex. Maybe mongoDB drops off line, promise reject is handled and node doesn’t stop (crash)
How do we handle unhandled promise exceptions?
In future, unhandled promises will crash application
Send a friendly error to client
log the exception on the server
How?
using promises - catch() rejected promise
using await/async - wrap in try/catch blocks
What is the problem with the current implementation?
How do we remove the try-catch blocks from our express async router handlers?
Hint: create a template function that takes route handler as an arg
create a middleware function (async) that serves as a template
pass route handler to middleware (encapsulates route in try/catch)
Why?
Make code less noisy
don’t have to repeat try-catch logic
Problem?
Makes route handler signature a bit noisy
Soln?
npm module use express-async-errors
monkey patches routers at run time (wraps route in try/catch)
How can we replace our error handling middleware function using an npm module?
Hint: express-async-errors
remove asyncMiddleware function
make routes cleaner
Mosh’s recommendation for handling express-async errors
What is winston?
In every enterprise application, must log errors
Why?
Determine where we can improve our application
What?
npm Winston
How do we log express errors in a file and in mongoDB?
Hint: only logs errors with express route handlers
npm i winston
npm i winston-mongodb
only catches errors that are part of request processing pipline
constrained to errors within express
How do we handle uncaught exceptions in a node process?
Hint: outside express framework
exceptions not caught in try/catch block
outside of express (route handlers)
process object is an event emiter
process.on(‘uncaughtException’, (ex) => {
winston.error (ex.message, ex);
});
ONLY WORKS WITH SYNCHRONOUS CODE
How do we handle unhandled promise rejections?
Hint: unhandled async promise rejections
process.on(‘unhandledRejection’, (ex) => {
winston.error(ex.message, ex);
})
What’s the problem with this implementation?
It violates separation of concern
not how enterprise applications are built
index.js should only orchestrate, details in separate modules
Ex. details of routes / database separate
Soln?
Create folder mkdir startup
add routes.js (file) - move routes to separate module
What are key questions about automated testing?
Hint: FAQs
- What is automated testing?
writing code to test the application code’s functionality, then running that test code in an automated fashion
- Does it replace manual testing?
- Do I really need it?
- How should I do it?
- Should I write test code first (TDD) or app code first?
6. What should I test in my application?
What is automated testing?
Source code consists:
Production Code - application APIs
Test Code - unit tests, integration tests, end to end tests
What is the problem with manual testing?
Hint: Many steps, slow workflow
Lot’s of steps!
Workflow is slow (takes minutes)!
As application grows, manual testing of every function (bit/peice) grows exponentially.
To test a function in the application manually:
Launch app
Login
Navigate
Fill out a form
Submit it
verify the result
**many steps, slow workflow, cannot test many functions
What are the benefits of automated testing?
testing code on a frequent basis, faster than manually
automated testing allows us to catch bugs before deployment
Deploy applications with confidence from testing by reducing number of bugs that go into deployment
Refactor code with confidence - retest behavior of application after refactoring
write higher quality functions - functions that work with various inputs and edge cases
What are the three types of automated tests?
Unit test
test a unit of the code without external dependencies
Integration test
tests a class or multiple classes with external dependencies
End-to-End Test
drive an application through its UI
What is unit testing?
Great for testing conditional statements and loops
use to test methods with complex logic and calculations
test a unit of the code indepent of external dependencies.
Cheap to write
Fast to execute
Can run hundreds in a few second
Can verify each building block
doesn’t give confidence in reliability of application
What is integration testing?
Test with external dependencies (web service, database, message queue)
Tests integration of application with external dependencies
slower to execute (read/write to database)
gives confidence to application
What is integration testing NOT?
a test that takes a few units (classes) and tests their behavior as a whole
Two classes tested together defined as an integration test vs. unit test
(even though not using external resources)
DO NOT DO:
Recipe for writing fragile tests (tests that break)
as implementation of classes changes, tests break
these tests slow you down
if you’ve failed at doing integration testing, you probably did it wrong
What is End-to-End Testing?
Drive an application through UI
tools for creating end to end tests (Selenium)
gives developers the greatest confidence
they’re very slow, brittle
Only test happy paths, leave edge case tests to unit tests
Ex. Selenium - record interaction of user with application, play back to check if app returns correct results
What is the test pyramid?
Hint: Unit test, Integration test, End-to-End test
Each application contains all testing catagories!
Ratio depends on each project
Testing should include:
Unit Tests
most tests in this catagory
easy to write, quick execution
gives little confidence
Integration Tests
a bunch to test integration of application code with external dependencies
gives greater confidence
benefits of end-to-end testing without user interface hassle
End to End Tests
few E2E tests
only test happy paths
leave edge cases to unit tests
What is a test framework?
Focus on writing good, clean, unit tests NOT the tooling
tooling comes and goes!
A Testing framework includes:
1. library
utility functions to write tests
2. Test running
a command line tool that runs tests and returns results
Popular testing frameworks:
Jasmine
first test framework, includes all testing needs
Mocha
most popular npm test framework
BUT requires 3rd party modules to cover all tests (chai, sinon)
requires different webpages for viewing documents
libraries may develop indepently and become incompatible in future
Jest
npm i jest –save/dev
wrapper around Jasmine + additional features
Mosh’s personal favorite
developed by Facebook
includes Jasmine + code coverage tool
used to test React apps
How do we install jest?
Hint: testing framework developed by Facebook
Wrapper around Jasmine
Developed by Facebook for testing React apps
npm i jest –save/dev
any file that ends with .js(x) treated like test file
update package.json “test”: “jest”
What is the testing naming convention?
moduleName.test.js
moduleName.spec.js
How do we write a test using jest?
How many unit tests per member?
cover all execution paths
cover logic thoroughly
Basic Guidelines:
number of unit tests >= number of execution paths
Ex. if number is positive, if number if negative, return 0
What are matchers in jest?
Hint: used to make assurtions
verify if a result is correct by comparing to another function (matcher)
Matchers for:
numbers
toBeLessThanOrEqual ( )
toBe ( )
toEqual ( )
toBeCloseTo ( )
strings
How can we trouble shoot problems in our functions?
Use jest error messages for testing / debugging
sometimes its a problem in production code
sometimes its a problem in test code
What becomes critical when writing unit tests?
organize tests so they are clean and mantainable
tests are first class citizens in source code
tests are as important as production code
always group related tests inside describe ( )
-a function in jest for grouping related tests
-clean up code by removing test keyword w/ it keyword
DO NOT:
write ugly, fat, unmaintable tests!
How does jest allow us to refactor a method with confidence?
refactor -
change implementation without changing behavior
refactor a method, re-run test to ensure nothing is broken
Ex.
How do we test strings using Jest?
Test shouldn’t be too specific or too general
How?
use a regular expression
search for certain patterns, not exact string matches
Ex. test a method that generates the title or body of an email
as long as name of person in string, okay (reg expression)
How can we use jest to test arrays?
Guidelines:
check for existance of element in array (irrespective of position)
Do not look for exact location of an element (as a test)
Do not look for exact array size
Example:
result. toBeDefined - too general
result. toBeNull - too general
result[0].toBe(‘USD’) - too specific, change sorting algo and it breaks
result.length.toBe(3) - too specific, breaks if add more elements
How can we test objects using Jest?
expect(result).toMatchObject
How do we test exceptions?
Do not get a result from exception
must pass a callback function
How do we continuously run tests?
Hint: have jest watch code/re-run tests after changes detected
Tedious to continually restart tests from terminal after every change
have jest monitor changes and re-run tests automatically
during production, keep one monitor with test code running / other with production code
jest -watch–all
How can we write a unit test for testing fizzBuzz?
How to unit test a function that directly or indirectly talks to an external resource?
Hint: polymorphism - override prototype member with mock function
replace real implementation of a function (calling external resource) with a fake “mock” implementation
in test, redefine external call (using a function)
override method on prototype with a new function (fake implementation)
How do we test the interaction of one object with another object?
Using flags
Ex. If notifyCustomer is called, mail.send will be called, our flag will become true
testing interaction of notifyCustomer object with mail object
better approach?
JEST mock functions
What is a Jest mock function?
Hint: replaces mock functions in vanilla javascript
helper functions (dummy functions)
can check arguments passed to functions
can easily check if a function has been called
used to replace interaction with outside dependencies
during unit testing (isolated test)
.toHaveBeenCalled()
.toHaveBeenCalledWith() - test method args (numbers, booleans, not string)
Ex. re-write mock functions using Jest mock functions
What is the unit testing rule of thumb?
use unit tests to test functions with algorithms
that have zero or minimum external dependencies
avoid creating too many mocks
too many mocks? use integration testing
What is ideal for unit testing?
generateAuthToken (user model)
no external resources
no mocking
simply call function and verify result
unit testing better than integration test
Why?
Don’t have to call an HTTP end point, excercise entire application stack to verify result
How can we unit test our generateAuthToken method?
What is integration testing?
testing our application with external resources
need a real database (populate with test data)
send HTTP request to server (database)
make an assertion (response or database)
Ex. Look at DB and verify new data is available
How do we setup the test db for integration tests?
Get db from config module
What is supertest?
Hint: npm module (integration testing)
send HTTP requests like using postman
How do we create our first integration test?
Hint: test api endpoint
supertest npm (module)
allows us to call API endpoints like we did manually with postman
Server best practice
load the server, close server each time
beforeEach ( ) - allows us to do something before each test (load server)
afterEach( ) - allows us to do something after (close server)
How do we integration test an API endpoint with parameters?
Hint: GET /:id
Cannot use
expect(res.body).toMatchObject(genre);
Why?
object_id
Soln?
Must use .toHaveProperty()
How do we write clean tests?
Hint: generateAuthToken ( ) repeated, happy path repeated
Define happy path
then, in each test, change one parameter
that clearly aligns with name of the test
How?
create a function for happy path (re-use)
setup test environment with beforeEach ( )
Ex. authentication testing
define happy path (post request to server)
change token (empty, invalid, valid)
How to test auth middleware (if a user sends a valid jwt, auth fn will decode jwt and pass to next in the request body)?
Hint: test if auth middlware function decodes JWT
Must use a unit test
create mocks (req, res, next):
- req = {header: jest.fn().mockReturnValue(token)}
- res = { }
3. next = jest.fn();
Ex. test auth middleware
create a user document (w/ id, admin props)
generateAuthToken based off user document
pass token to auth fn w/ mocks (res, next)
compare auth fn output w/ user document (test if auth decodes JWT)
What is a code coverage tool?
Hint:
What scenarios have you not tested?
Shows us how much of our code
How much of code is covered by tests?
What scenarios have you not tested?
Use Jest to see how much code has been covered
Package.Json:
jest –watchAll –verbose –coverage
How can I know how much of my code is covered?
Hint: Jest Coverage Report
What are scenarios that haven’t been tested?
How much of code is covered by tests?
Why?
Unit and Integration Testing allow us to have greater confidence when deploying our applications
Test Frameworks allow us to write tests that verify behavior before deploying our applications
This allows us to deploy reliable products with less bugs
Where?
coverage folder
index.html file
Launch with live server
How can we read the JEST code coverage report?
Tells us where we are low on testing (more tests needed)
Ex. Admin.js needs more testing
% covered
not explicit coverage, % of code in file has been excercised by one or more tests
How can we specifically see what code is not covered by tests in jest coverage reporting?
Click on the file name
highlighted red is not covered by tests
Ex. genres.js - GET /:id
if invalid objectID, validateObjectId returns 404
BUT
if valid objectID, no test for “genre not found”
SOLN
add test
What is test driven development? (TDD)
Hint: write tests before production code
An approach to writing software
Development is driven by tests:
Write tests first, production code second
Foundation of test driven development:
Start, by writing a failing test (no application code)
Next, write simplest application code to make it pass (pass test)
Finally, refactor code if necessary
Lastly, repeat until feature is built
Why?
Don’t have to change code or re-work code to test it
Source code is testable immediately
Deploy with confidence
Every line of production code is covered by tests
Simplier solutions
reduce over engineering complexity
write simplest code to make tests pass
What are the benefits of test driven development?
Hint: write tests before production code
Don’t have to change code or re-work code to test it
Source code is testable immediately
Deploy with confidence
Every line of production code is covered by tests
Simplier solutions
reduce over engineering complexity
write simplest code to make tests pass
How can we add the ability to return a movie using test driven development?
How do we send a request to our server to return a movie?
Rental model has properties...
customer
movie
date out (undefined)
date returned (undefined)
rental fee (undefined)
When customer returns movie, we set return date and calculate payment on server
Approach?
Cannot update rental document (don’t want client to set rental fee, our date out)
create a new endpoint for returning a movie
‘/api/returns’
send return request (add customerId and movieID)
will look up rental, set return date / calculate rental
How do we use TDD (test driven development) to build returns API?
Ex: //return 401 if client is not logged in
Write test first
by default, will fail if no production router exists
write production router until test passes
How to do we use TDD to define test cases for sending HTTP client requests to our server?
Hint: validate client data
add joi later (during refactoring)
write clean tests
What’s the next step in using TDD (test driven development) to complete the returns api feature?
What is the next step in designing our feature using TDD?
Test development:
change one variable
execute happy path
analyze results
Feature development:
simplest implementation
refactor later
How can we refactor this code?
Add our auth middleware now
replace res.status(401).send(‘Unauthorized”)
add middleware (refactor)
How do we use TDD to create the dateReturned functionality in our api?
verify a property was set and saved in db
How can we refactor this test?
How can we refactor this route handler?
Beauty of TDD (test driven development)
100% of router is covered by tests
Can refactor and re-run tests
1. Add Joi - refactor to validate request using joi
How do we refactor this .findOne method?
create lookup as a static method (available on Rental class) in the rental model
Ex.
Static: Rental.lookup()
instance: new User().generateAuthToken()
How should we refactor this route handler?
handler is too busy calculating what should happen with state of rental object
information expert principle
logic to modify state of an object should be encapsulated with object
Soln?
encapsulate logic in rental object (calculating rental fee, return date)
What is 404 HTTP error code?
404 not found
resource requested is no longer available
What is 403 HTTP error code?
403 forbidden
server has rejected your request
What are the two deployment options for node applications?
Hint: Paas vs. Docker
Platform as a Service (PaaS)
Heroku, Google Cloud Platform, AWS, Azure
great option if you don’t want to get involved with infrastructure
if you don’t want to worry about servers, load balancers, reverse proxies, restarting application on crash
Docker
if you want to deploy to your own web servers
can create image of application and deploy to any computers you want
$How do we prepare our application for production?
Install a few packages
npm i helmet
middleware package to protect app from well known vulnerabilities
protects headers and client side attributes from attacks
npm i compression
compress HTTP response we send to client
What is heroku?
How to prepare app for deployment to heroku?
add to package.json
“scripts”: {
“test”: “jest –watchAll –verbose –coverage “,
“start”: “node index.js”
},
“engines” : {
“node” : “12.18.2”
},
How can we push changes from our local repository to our Heroku repository?
Hint: Git remote
> Heroku create
Two things:
creates a git repository in Heroku
connects git remote to local git respository
When we push changes to our local git repository
Heroku will automatically download changes to our application code (in local repositry)
How do we push our local changes to heroku?
Heroku create
git push heroku master
pushes code from local git repo to heroku repo
How do we view error logs in our application on Heroku?
through the terminal
through the dashboard
How do we set environment variables for our app using heroku?
heroku config: set vidly_jwtPrivateKey=123
heroku config
*in heroku, all dynos (servers) share environment variables
How can we add a database to our heroku deployment?
store db password in environment variable
heroku config:set vidly_db=mongodb+srv://vidly:1234@vidly.lmsvf.mongodb.net/?retryWrites=true&w=majority
