Pivotal Engineering Journal

Technical articles from Pivotal engineers.

  1. Faking OAuth2 Single Sign-on in Spring, 3 Ways When your Java Spring web application depends on a third-party OAuth2 single sign-on service, tests can be slow, brittle, or difficult to control. In this article we’ll show you three ways to address these issues by faking OAuth2 single sign-on in your tests. Categories:   Spring    Java    Testing    WireMock    OAuth2   
  2. Flow Your Tests Like Your Code When writing Behavioral Driven Development (BDD) Golang unit tests, pattern the flow of the tests after the code; it will make the tests easier to understand, navigate, and maintain. Categories:   Golang    TDD    BDD   
  3. How To Enable IPv6 on Your Cloud Foundry's HAProxy This blog post describes how to assign both an IPv4 and an IPv6 address to the HAProxy VM (an optional load balancer) on a Cloud Foundry deployment (foundation). This may be of particular interest to homelab enthusiasts who have an abundance of public IPv6 addresses but only one public IPv4. Categories:   BOSH    CF    IPv6    vSphere   
  4. Scaling the Firehose API and Consumers If you’re still losing logs even though you’ve correctly scaled Dopplers, you probably need to scale your Firehose API and consumers. Categories:   CF    Cloud Foundry    PAS    Pivotal Application Service    Loggregator    Logging & Metrics   
  5. Scaling Doppler VMs in Cloud Foundry When your Cloud Foundry logging system is not working optimally, one of the reasons could be improperly scaled Doppler VMs. Read this learn if this is the case and get general guidelines for how to properly scale your Doppler VMs. Categories:   CF    Cloud Foundry    PAS    Pivotal Application Service    Loggregator    Logging & Metrics   
  6. ESM Pairing Study Data Anonymized Data from a pairing research study. Categories:   Research    Agile    Pair Programming    Pairing   
  7. Benchmarking Apache Geode's Performance Measuring and Improving Performance for a Faster Geode 1.10 Categories:   Performance    Apache Geode    AWS    Java    Testing   
  8. How we built a controller using KubeBuilder with test-driven development, Part 2 Lessons we learned while building a controller with KubeBuilder, and how we made our unit tests fast enough for test driving. Categories:   Greenplum    Kubernetes    KubeBuilder    Test Driven Development   
  9. How we built a controller using KubeBuilder with Test Driven development, Part 1 Lessons we learned while building a controller with KubeBuilder Categories:   Greenplum    Kubernetes    KubeBuilder   
  10. ESM Study Data Anonymized Data from a research study. Categories:   Research    Agile   
  11. Merge Join Support In GPORCA We look at how FULL JOINS can be supported natively in GPORCA, allowing GPORCA to generate plans providing better query execution times. Categories:   Databases    Query Optimization    SQL    Greenplum    GPORCA    Full Join   
  12. Greenplum: Speeding up JDBC Reads in PXF Using Partitioning Parallel data reads from external databases in PXF using JDBC Partitioning Categories:   Greenplum    PXF    JDBC    Partitioning    Parallel    JDBC Reads   
  13. Administering a PXF cluster using the PXF Command Line Interface The PXF CLI provides a convenient way to administer PXF servers running on Greenplum Categories:   Greenplum    GPDB    PXF    CLI    Utilities    Amazon S3    HDFS   
  14. Fast Access to Your S3 Data with Greenplum PXF Greenplum’s Platform Extension Framework (PXF) supports querying data on AWS S3. PXF now supports AWS S3 Select which will enable Greenplum users to run their queries on data stored in AWS S3 more efficiently with up to 4x performance improvements. Categories:   Greenplum Database    GPDB    Platform Extension Framework    PXF    S3 Select    AWS    Amazon S3    Amazon Simple Storage Service   
  15. Ensure Windows apps are using the `windows` stack Pivotal Application Service for Windows introduced the -s windows stack name in PASW 2.4, reducing the operator and developer need to concern themselves with specific Windows Server versions. From PASW 2.4 onwards, both the windows2016 and windows stack names worked - giving sufficient time to migrate apps over to the new stack name. Categories:   PASW    .NET    Windows    stack    cf   
  16. A High-performing Mid-range NAS Server, Part 3: 10 GbE Upgrading our iSCSI (Internet Small Computer System Interface) NAS (network-attached storage) server from 1 GbE (gigabit ethernet) to 10 GbE increased our vSphere VM’s (Virtual Machine’s) sequential read throughput 43%, sequential write throughput 940% (!), and IOPS (Input/Output Operations Per Second) 60%. Categories:   Logging & Metrics    FreeNAS   
  17. How we moved a massively parallel Postgres database onto Kubernetes Distributed applications that scale out, such as Greenplum, fit well with Kubernetes. Categories:   Greenplum    Kubernetes    PostgreSQL    Databases   
  18. Testing Spring filters without pain This article aims at showing you how to test http filters effectively, by avoiding unnecessary mocking and rather appropriately using the test toolkit that Spring offers. Categories:   Spring    TDD    Mocking    Kotlin   
  19. Eureka, Zuul, and Cloud Configuration - Pivotal Cloud Foundry Example and explanation of how to set up a common use case with the Spring Cloud Netflix stack within a PCF deployment. Categories:   Java    Spring    Annotations    Spring Boot    Pivotal Cloud Foundry   
  20. Eureka, Zuul, and Cloud Configuration - Local Development Example and explanation of how to set up a common use case with the Spring Cloud Netflix stack to prototype for local development. Categories:   Java    Spring    Annotations    Spring Boot   
  21. Using Greenplum to access Minio distributed object storage server Parallel data transfer data between Greenplum and Minio Categories:   Greenplum    Minio    PXF   
  22. Transferring Time-based One-time Passwords to a New Smartphone Time-based One-time Passwords (TOTP) authenticator apps are often deployed on smartphones to enhance security of sensitive online accounts; however, replacing the phone typically requires a burdensome reset of all TOTP passwords. In this blog post we describe the clever use of a QR code reader, secure storage, bash scripting, and a QR code generator to quickly configure a new phone (no reset of TOTP required). Categories:   2FA    TOTP    Google Authenticator   
  23. Managing Stateful Apps with the Operator Pattern; Orchestration Considerations Stateful Kubernetes apps can be managed with the Operator Pattern, and have other considerations regarding their orchestration; the forth blog of a series of 4 on Stateless Kubernetes Apps
  24. Storing Stateful Data that Outlives a Container or a Cluster; Optimizing for Local Volumes Storage for the data of stateful Kubernetes apps must outlive a Kubernetes container or a cluster, and can be optimized with Local Volumes; the third blog of a series of 4 on Stateless Kubernetes Apps
  25. Provisioning Stateful Kubernetes Containers that Work Hard and Stay Alive How to develop Kubernetes containers that resist eviction, the second blog of a series of 4 on Stateless Kubernetes Apps
  26. Stateful Apps in Kubernetes Defining stateful versus stateless applications in terms of Kubernetes service resiliency. The first blog post of a series of four about Stateful Kubernetes Apps.
  27. Troubleshooting Obscure OpenSSH Failures How an elusive CI (Continuous Integration) error led us to uncover a hidden man-in-the-middle ssh proxy. Categories:   Networking   
  28. Greenplum for Kubernetes Operator Enabling Declarative Style Greenplum Deployment in Kubernetes Categories:   Greenplum    Kubernetes    Agile   
  29. Understanding Keep-Alive Timeouts in the Cloud Foundry Networking Stack Let’s journey through a debugging session to understand best practices in configuring the Cloud Foundry deployment’s networking stack, debug interacting keep-alive timeouts, and get familiar with the TCP layer. Categories:   Networking    Routing    Cloud Foundry    Keep-Alives    Debugging    TCP   
  30. Quoted and Unquoted Template Variables in Concourse CI How to use quoted and unquoted variables in Concourse CI templates Categories:   Concourse   
  31. Development of Greenplum for Kubernetes using Minikube Speed up agile developing process using a minikube Categories:   Greenplum    Kubernetes    Agile   
  32. Concourse as a Service and Multi-CPI Leveraging a singular BOSH director to manage Concourse for multiple teams Categories:   BOSH    Concourse    GCP    Infrastructure   
  33. Let's Contribute to Golang! How to start contributing to the Golang project with an example Categories:   Golang    contributing    OSS   
  34. Safely Upgrading PAS 2.2 with NSX-T Load Balancers How to upgrade an NSX-T-based PAS 2.2 → 2.3 foundation without downtime Categories:   NSX-T    Operations Manager    BOSH   
  35. GiST Support In GPORCA We look at how GIST indexes can be supported in GPORCA, allowing GPORCA to generate plans providing better query execution times. Categories:   Databases    Query Optimization    SQL    Greenplum    GPORCA    GiST   
  36. Frontend Contract Tests Without Magic Numbers Banish magic numbers from your frontend contract tests! Categories:   Spring    API    Contract Testing   
  37. Let's use Vault - Part 3: Setting up Vault How to configure Vault for your CI Categories:   GCP    Kubernetes    Vault    Credhub   
  38. Let's use Vault - Part 2: Setting up Vault How to configure Vault for your team Categories:   GCP    Kubernetes    Vault    Credhub   
  39. Let's use Vault - Part 1: Deploying Vault How to deploy Vault on Kubernetes using Google Cloud Storage as its backend Categories:   GCP    Kubernetes    Vault    Credhub   
  40. Diagnosing Ruby Memory Issues in Cloud Foundry's API Server How the Cloud Foundry CAPI team debugged Ruby memory usage issues in the Cloud Controller API Categories:   Cloud Foundry    CF    Cloud Controller    Ruby    Ruby Memory Management   
  41. How to use "tag_filter" in "git" resources in Concourse CI How to use “tag_filter” in “git” resources in Concourse CI Categories:   git    GitHub    Concourse   
  42. How to Install a TLS Certificate on vCenter Server Appliance (VCSA) 6.7 We install a Transport Layer Security (TLS) certificate issued by a commercial Certificate Authority (CA) on a VMware VCSA 6.7 while avoiding several pitfalls. Categories:   vSphere   
  43. How Pivotal contributes to the Development of PostgreSQL 11 Summary of Pivotal Contributions to PostgreSQL 11 Categories:   Greenplum Database    PostgreSQL    Greenplum    Development   
  44. Create Regression Tests for Greenplum Database How to create new regression tests for Greenplum Database Categories:   Greenplum Database    Databases    Testing    Regression Tests   
  45. Windows Containers in Cloud Foundry? Here's How We Did It Technical details for how we implemented containers on Windows Server Core 2016 for Pivotal Cloud Foundry. Categories:   CF Runtime    Diego    Windows    Containers    .NET   
  46. Scaling Machine Learning to Recommend Driving Routes
    The Pivotal Data Science Labs helped a multinational customer build a scalable, real-time predictions and recommendations application to increase revenue. We built an end-to-end machine learning workflow which addresses online deployments and offline training using open source projects and Pivotal products.
         Categories:   Spark & Hadoop    real-time prediction & recommendation    feature engineering & modeling pipeline    driver revenue prediction    Scale Machine Learning as a service   
  47. Failure is a part of the game Failure is necessary
  48. Benchmarking the Disk Speed of IaaSes It’s helpful to know the performance characteristics when selecting a disk type for an Infrastructure as a Service (IaaS). In this blog post we describe the results of benchmarking various disk types of various IaaSes, including Amazon Web Services (AWS), Microsoft Azure, Google Compute Engine (GCE), and VMware vSphere. We measure Input/output operations per second (IOPS), read throughput, and write throughput. Categories:   BOSH    Logging & Metrics   
  49. Event Sourcing with Kafka, RabbitMQ and JPA Demonstrating Event Sourcing with various methods using Kafka, RabbitMQ and JPA Categories:   Event Sourcing    Kafka    RabbitMQ    JPA    Spring Cloud Stream   
  50. Simplify Deploying Kubernetes with Cloud Foundry Container Runtime Cloud Foundry Container Runtime makes deploying Kubernetes easy with the power of BOSH. Categories:   CFCR    BOSH    Tutorial    Kubernetes    DevOps   
  51. Deploying BOSH VMs with IPv6 Addresses on vSphere Recent changes to the BOSH software suite enable the assignment of IPv6 addresses to VMs deployed by the BOSH Director in a vSphere environment. In this blog post we describe how we deployed a BOSH Director and subsequently used the Director to deploy a web server with a private IPv4 address and a public IPv6 address. Categories:   BOSH    IPv6   
  52. Diagnosing performance issues with Java legacy apps on Pivotal Cloud Foundry How to profile apps on Pivotal Cloud Foundry. VisualVM to the rescue! Categories:   Profiling    Java    VisualVM    Performance    CPU Sampling   
  53. All I do is VIM VIM VIM A subset of useful vim commands for beginners Categories:   vim   
  54. Maintaining BOSH Directors with Concourse CI and bosh-deployment BOSH directors can be deployed (and redeployed) with a properly built Concourse pipeline and manifests generated by the curated bosh-deployment manifest generation tool. This lightens the burden of keeping BOSH directors patched & up-to-date. Categories:   BOSH    Concourse   
  55. Injecting Credentials Into Spring Boot Apps – The Easy Way Configuring Spring Boot apps in Cloud Foundry is super easy once we take advantage of ConfigurationProperties and user-provided services. Categories:   Spring Boot    Cloud Foundry    CF   
  56. React integration testing with Enzyme We used Enzyme, a popular React unit testing library, for full-on frontend integration tests. Find out how our setup greatly improves feedback and avoid the pitfalls we’ve come across. Categories:   React    Enzyme    Javascript    Testing   
  57. Testing Greenplum Database using fault injection How to test Greenplum Database using fault injection Categories:   Greenplum Database    Greenplum    Testing frameworks    Testing   
  58. Continuous Market Disruption This blog post explores how enterprises can reduce waste and continuously experiment leading to the product evolution and effective use of CAPEX to realize the true ROI Value of cloud native with Pivotal. Categories:   App Transformation    Replatforming    Modernization    Pivotal Cloud Foundry    Cloud Native ROI Continuum    Agile    XP   
    Connecting To PCC with Spring Boot How do you get started with Pivotal Cloud Cache (PCC)? How do you connect your spring boot application to PCC? This blog post attempts to provide you an example on how to connect your spring boot application to PCC. Categories:   PCC    CF   
  60. Interpreting Decision Trees and Random Forests This blog dives deeper into the fundamentals of decision trees and random forests to better interpret them. Categories:   Decision Tree    Random Forest    Data Science   
  61. Transitioning to CredHub on Pivotal Web Services (PWS) CredHub is designed to store passwords, keys, certificates, and other sensitive information for a BOSH-managed environment. Pivotal’s Cloud Operations (CloudOps) team recently migrated credentials for PWS to CredHub. Here’s how we did that. Categories:   CredHub    Operations    Cloud Foundry    CF   
  62. Day in the Life - Santa Monica Style A day in the life of a Pivotal Cloud Foundry software engineer in Santa Monica, California. Categories:   BBL    Cloud Foundry   
  63. Deploying GrootFS to Pivotal Web Services (PWS) Our rollout of GrootFS to Pivotal Web Services was a gradual, iterative process that allowed us to test on a small subset of production instances, roll the changes back, make improvements, and finally release it with confidence. We talk about the process and provide takeaways for other teams deploying new features to production. Categories:   Operations    Rollout    Cloud Foundry    CF    Agile   
  64. Elixir Clustering And Discovery On Cloud Foundry A tutorial for configuring an Elixir application to automatically cluster on Cloud Foundry (eg. PWS) using container to container networking. Categories:   Elixir    Cloud Foundry   
  65. Deploying a BOSH Director With SSL Certificates Issued by Commercial CA A BOSH director is typically deployed with self-signed SSL (Secure Sockets Layer) certificates; however, the director can be deployed with certificates issued by a trusted CA (Certificate Authority). Here’s how. Categories:   BOSH   
  66. Greenplum and Apache Spark via JDBC Using Greenplum and Apache Spark via JDBC in 5 minutes Categories:   Greenplum    Apache Spark    JDBC    Postgresql   
  67. Hibernate to JDBC An exploration in replacing Hibernate with JDBC in Spring. Categories:   Spring    Java    Hibernate    JDBC    Java Database Connectivity    Repositories   
  68. sql_magic: Jupyter Magic to Write SQL for Apache Spark and Relational Databases An IPython library to help data scientists write SQL code Categories:   Data Science    Jupyter Notebook    SQL    Greenplum    Apache Spark   
  69. Half a Million Concurrent/Distributed JMeter Users with BOSH ... in 10 Minutes This post details the steps of utilizing BOSH to setup half a million (potentially much more) concurrent and distributed Apache JMeter threads for load testing. Categories:   BOSH    Apache JMeter    Distributed Load Testing    Distributed Stress Testing   
  70. Using Terraform, Concourse, and om to Continuously Deploy Pivotal Cloud Foundry's Elastic Runtime A look at how Release Engineering deploys and tests the Elastic Runtime Tile Categories:   Infrastructure    Elastic Runtime Tile    Release Engineering    Terraform    Ops Manager    om   
  71. A Smart Editor and Visualizer for BOSH Manifests BOSH Editor is in-browser application that facilitates the creation of BOSH manifests. It includes smart auto-complete features and a way to visualize different segments of the manifest on the fly. Categories:   BOSH    Editor    Visualization   
  72. The Test Double Rule of Thumb A guide to understanding and effectively utilizing test doubles. Categories:   Agile    TDD   
  73. Deploying a Docker Registry to Cloud Foundry Pushing a Docker Registry to Cloud Foundry can be a powerful approach to distributing private Docker images to teams with minimal overhead. Categories:   Docker    Cloud Foundry   
  74. Testing for Data Consistency on Cloud Cache using BOSH and Turbulence Testing for data consistency on the Cloud Cache team using BOSH and Turbulence Categories:   BOSH    Turbulence    Cloud Cache   
  75. dep Coming to Unify Package Management in Go Golang’s future standard package manager is becoming more usable everyday. Here’s why it’s necessary and how you can try it out today. Categories:   Golang    Package Management    dep   
  76. Using Luigi Pipelines in a Data Science Workflow This post shows how we use Luigi as a pipeline tool to manage a data science workflow. We walk through an example analyzing network traffic logs. Categories:   Data Science    Luigi    Greenplum    Python    SQL   
  77. Must-Know Spring Boot Annotations: Controllers Learn about the most essential, must-know annotations for Spring Boot controllers. Categories:   Spring    Spring Boot    Annotations    MVC   
  78. Scoring at Scale with Keras and TensorFlow on Greenplum How to train a deep neural network with Keras and TensorFlow and then apply it for scoring on Greenplum. Categories:   Data Science    Greenplum    Greenplum Database    SQL    Python   
  79. The gpfdist protocol for External Tables in Greenplum Database The internals of the gpfdist protocol used for External Tables in Greenplum Database Categories:   Greenplum Database    Databases   
  80. Deploy To vSphere NSX-T Opaque Networks Using BOSH BOSH now allows attaching vSphere deployed VMs to NSX-T’s Opaque Networks Categories:   BOSH    BOSH CPI    vSphere    NSX-T   
  81. Visualizing Cloud Foundry with Weave Scope - Part 1 This is the first of two blog posts showing you how Weave Scope, a visualization and troubleshooting tool originally aimed at Docker and Kubernetes, can be used to reveal the hosts and network topologies for arbitrary BOSH deployments. As such, Scope is able to visualize the many components that make up Cloud Foundry and the interplay between them. This gives everyone, from newcomers to seasoned experts, new ways to learn about Cloud Foundry and troubleshoot a running system if necessary. Categories:   Cloud Foundry    CF    BOSH    Weave Scope    Visualization    Distributed Systems   
  82. Testing in Swift with Dependencies Testing against frameworks/libraries is tricky in Swift because we can’t just spy on dependencies and fake out the response. Here is how we test neatly in Swift 3. Categories:   Swift    Swift 3    Protocol    Testing   
  83. Plotting Using an MPP Database A tutorial on how to build histograms, scatter plots, and ROC curves using an MPP database and plot them in Python or R. Categories:   HAWQ    GPDB    Big Data    PostgreSQL    Postgres    MPP    Histogram    Scatter Plot    ROC Curve   
  84. BOSH + Apache JMeter(TM) = Tornado for Apache JMeter(TM) Tornado for Apache JMeter™, is a BOSH release for Apache JMeter™. It simplifies the usage of JMeter™ in distributed mode, making it easier to perform load or stress testing. Categories:   BOSH    Apache JMeter    Distributed Load Testing    Distributed Stress Testing   
  85. Git pushInsteadOf Best practices for Git on mixed-source teams Categories:   git   
  86. Spring for Normal People Learn how to Spring Boot the easy way with TDD and Thymeleaf. All the gain, half the pain! Categories:   Spring    Spring Boot    TDD    Humans   
  87. TDD with React and MobX A look into testing MobX + React, plus why MobX is a viable alternative to Redux. Categories:   TDD    React    MobX    Redux    Javascript   
  88. Trilogy and Greenplum for Data Science TDD How to use a new SQL testing framework called Trilogy with Greenplum Database to help you test drive your data science code. Categories:   Data Science    TDD    SQL    Databases    Greenplum Database   
  89. Trilogy - the database testing framework A quick overview of a new database-agnostic SQL testing framework Categories:   SQL    TDD    Testing frameworks    Databases   
  90. Continuous Integration for Data Science This article will show why continuous integration is also important for smart apps projects. Categories:   Data Science    Machine Learning    Smart Apps    TDD    Continuous Integration    Concourse   
  91. Agile Development for Highly Scalable Data Processing Pipelines Legacy data processing pipelines are slow, inaccurate, hard to debug, and can cause thousands of dollars in revenue. Conforming to agile methodology and a detailed seven-step approach can ensure an efficient, reliable and high-quality data pipeline on distributed data processing framework like Spark. Learn how following TDD, careful creation of data structures, and parallel execution results in: code competency and completeness, and a linearly or constantly scalable robust big data processing pipeline. Categories:   Linear-Scale    Data-Pipeline    Agile-TDD    SPARK    Pair-programming   
  92. Why Is My NTP Server Costing $500/Year? Part 3 Running a Network Time Protocol (NTP) server in the pool.ntp.org project can incur $500/year in data transfer (bandwidth) costs. Those costs can be reduced or even eliminated by choosing alternative Infrastructure as a Service (IaaS) providers, modifying the server’s pool.ntp.org connection speed setting, choosing an alternative continent upon which to place the server, and modifying the NTP daemon’s configuration file to rate-limit the clients. Categories:   NTP   
  93. The File protocol for External Tables in Greenplum Database The internals of the File protocol used for External Tables in Greenplum Database Categories:   Greenplum Database    Databases   
  94. Enforcing Separation of Concerns with Declarative Programming Using a popular academic program, FizzBuzz, and an extreme declarative programming approach using Datalog, we examine the thought process and benefits for enforcing separation of concerns with declarative programming. Categories:   Declarative Programming    datalog   
  95. Understanding and Mitigating the .io Top-Level-Domain failure in Cloud Foundry and Pivotal Web Services An article about the .io TLD failure, how it affected Cloud Foundry, as well as how we could potentially mitigate TLD failures in the future. Categories:   CF Runtime    DNS    Cloud Foundry    Pivotal Web Services   
  96. Public IPs for diego cells On September 29th, 2016, Pivotal Web Services (PWS) enabled a feature available in BOSH to auto-assign public IPs to Diego cells. Categories:   BOSH    CF Runtime   
  97. Using the beta BOSH CLI to Deploy an IPv6-enabled nginx Server to AWS Amazon Web Services (AWS) has recently announced Internet Protocol version 6 (IPv6) Support for their Elastic Compute Cloud (EC2) Instances in Virtual Private Clouds (VPCs). In this blog post, we describe using the beta BOSH command line interface (CLI) to deploy a virtual machine (VM) running nginx (a popular webserver) to EC2 with a native IPv6 address. Categories:   BOSH    IPv6   
  98. Using docker-image-resource to build a custom container for testing your Ruby apps in Concourse Sometimes we want to create custom docker images with external dependencies cached. Learn how to have Concourse automate this and use the built container to run tests. Categories:   Concourse    Docker    Ruby    Containerization   
  99. Cloud Foundry GrootFS: A daemonless container image manager GrootFS is the new container image plugin for Garden, Cloud Foundry’s container engine. It doesn’t require a daemon process, as most other engines, and can be run as an unprivileged user, improving CF’s security posture. Categories:   Cloud Foundry    Containers    Docker    Community    Open Source    Linux    Garden   
  100. Don't mix goroutines and namespaces: Part 1 Hands on with Linux namespaces and threads Categories:   containers    linux   
  101. Headless UI Testing with Go, Agouti, and Chrome Acceptance tests for your UI are an excellent way to cover user functionality. Let’s see how to write a simple acceptance test in Go with Agouti and have it run headlessly in a CI environment with Chrome. Categories:   Testing    Go    Agouti    Chrome    UI Testing    UI   
  102. Leveraging NSX's Features with BOSH's vSphere CPI BOSH, a VM (Virtual Machine) orchestrator, includes the ability to interoperate with NSX, a network virtualization platform, when deploying to a vSphere IaaS (Infrastructure as a Service). This blog post describes deploying VMs as the backend of an NSX Load Balancer. Categories:   BOSH    NSX    vSphere   
  103. Profiling Query Compilation Time with GPORCA GPORCA is Pivotal’s Query Optimizer for Greenplum Database and Apache HAWQ (incubating). In this post, we describe how users can profile query compilation with GPORCA. This will aid users in understanding which of GPORCA’s steps is the most resource intensive, and what transformations are being triggered. Based on this information, users can provide query hints to reduce or increase the search space, see where the time and memory is being spent, and learn how to influence its decision making. Categories:   Databases    GPDB    HAWQ    PQO    GPORCA    Query Optimization    SQL   
  104. Everything and the Spring Cloud Data Flow Sink Get started with Spring Cloud Data Flow Streams by creating a custom Sink app and deploying to Pivotal Cloud Foundry. Categories:   Spring Cloud Data Flow    Spring Cloud Data Flow Streams   
  105. How to Customize a BOSH Stemcell BOSH Stemcells are Linux-based bootable disk images upon which BOSH applications may be deployed. This blog post describes a process to customize a stemcell (most often used to troubleshoot stemcell boot problems). Categories:   BOSH   
  106. Updating a BOSH Release Authors of a BOSH Release may want to release a new version when the upstream application is updated. This blog post describes the process of updating a BOSH Release while avoiding common pitfalls. Categories:   BOSH   
  107. Test-Driven Development for Data Science Unravelling Test-Driven Development for Data Science. Categories:   TDD    Data Science    Machine Learning    Agile    Pair Programming   
  108. How to Set up a Distributed Elixir Cluster on Amazon EC2 Learn how to set up an Elixir cluster and how to deploy a Phoenix application on Amazon EC2. The techniques outlined in this article can equally apply to other providers such as Digital Ocean and Linode. Categories:   Elixir    Distributed    Cluster    Phoenix    Edeliver    Deployment    Erlang   
  109. Improving Query Execution Speed via Code Generation A code generation based solution inside the GPDB execution engine. Categories:   GREENPLUM DATABASE    QUERY EXECUTION    CODE GENERATION   
  110. Concourse has Badges Use Concourse’s badges to display the health of your project Categories:   Concourse   
  111. Concourse without a Load Balancer nginx is a less-expensive alternative to a load balancer for a BOSH-deployed Concourse server’s SSL termination. Categories:   BOSH    Concourse   
  112. GPDB merge with PostgreSQL 8.3 Greenplum merge with PostgreSQL 8.3 Categories:   PostgreSQL    Greenplum Database    Databases   
  113. Operationalizing Data Science Models on the Pivotal Stack Categories:   Data Science    Greenplum    SCDF    PCF    GemFire   
  114. Writing an Ionic 2 Application for Production On a recent Pivotal Labs engagement, we built a hybrid application using Ionic 2. This post provides some technical considerations for building your own. Categories:   Mobile    Hybrid Apps    Ionic   
  115. Faster Pipelines With Compiled BOSH Releases Compile once, deploy many times. Categories:   BOSH    Concourse    Testing   
  116. Improving Constraints In ORCA ORCA is Pivotal’s Query Optimizer for big data. We look at how we improved ORCA’s understanding of logical constraints. Categories:   Databases    Query Optimization    SQL   
  117. Creating a Custom Buildpack This article will describe how to create a custom buildpack using Rust as an example language. Categories:   Cloud Foundry    Buildpacks    Rust    CF Runtime   
  118. API First for Data Science How API first can help to create smart data-driven apps. Categories:   Data Science    Machine Learning    API First    Cloud Foundry    Smart Apps   
  119. Building a Native Navigation Menu for iOS with Turbolinks 5 Using Turbolinks 5 to hide your site’s HTML navigation and present a native navigation view to mobile app users. Categories:   Rails    Turbolinks    iOS   
  120. Using Action Cable With Cloud Foundry A guide to configuring and deploying a Rails 5 Action Cable app to Cloud Foundry. Categories:   Ruby On Rails    Cloud Foundry   
  121. Why you should stub, not shallow render, child components when testing React A better way to avoid brittle unit tests in React Categories:   TDD    React    Javascript   
  122. Data Driven Testing with Spek Finding a usable approach to data-driven testing with Spek Categories:   Kotlin    TDD    Spek   
  123. Virtual memory settings in Linux - The Problem with Overcommit How to tune the Memory Overcommit settings in Linux Categories:   Linux    Greenplum Database    Virtual Memory    Overcommit   
  124. Running GPDB using Docker A look into how the GPDB R&D team uses Docker to increase development consistency. Categories:   GPDB    Docker   
  125. SQL Stored Procedure Versioning Strategy A versioning strategy for SQL stored procedures provides flexibility for developers both on the DB and the application side. Categories:   SQL    Version control   
  126. SQL Test Driven Development with Oracle RDBMS Test-driving SQL stored procedures using Oracle SQL Developer IDE. Categories:   SQL    TDD    Oracle   
  127. Java Deserialization, JMX and CVE-2016-3427 If you use remote JMX, you need to update your JVM to address CVE-2016-3427 Categories:   Security    Java    Apache Tomcat    Pivotal tc Server   
  128. ByteA versus TEXT in PostgreSQL (for textual data) One of our customers switched from MongoDB to PostgreSQL, and the migration tool created all data fields as ByteA instead of TEXT. Makes one wonder, if there is a performance difference and if TEXT could be a wiser choice. Categories:   PostgreSQL    Performance   
  129. How to Deploy a Multi-homed BOSH Director to a vSphere Environment We explore deploying a multi-homed BOSH Director to a vSphere environment to segregate networks in a secure manner. Categories:   BOSH   
  130. TDDing React + Redux Helpful patterns for unit testing a React-Redux app Categories:   TDD    React    Redux    Javascript   
  131. Using Postgres to analyze ride data Postgres provides some fantastic functionality to help out with basic data analysis. This article will show you how to generate leaderboards and find streaks in raw sql data. Categories:   PostgreSQL    SQL    Databases   
  132. Distributed Pair Programming: What Works! Tales of pair programming on a distributed team. Categories:   Agile    Pair Programming   
  133. Implementing Containers on Windows: A Deep Dive Technical details for how we implemented containers on Windows for Pivotal Cloud Foundry. Categories:   CF Runtime    Diego    Windows    Containers    .NET   
  134. Testing JavaScript's native Promises A straightforward look at how to apply test-driven development to native JavaScript promises. Categories:   TDD    JavaScript    Promises    Testing   
  135. Running Tests in AWS Lambda Quickly and easily run your tests on AWS without the hassle of starting new EC2 instances. Categories:   AWS Lambda    Testing   
  136. ETL Journey from Oracle to Postgres How we transferred a legacy Oracle database to a new Postgres database in a 3 hour window. Categories:   ETL    oracle    postgres    database   
  137. SERIAL Datatype Performance in Greenplum Database How to improve the performance of the SERIAL datatype in Greenplum Database Categories:   PostgreSQL    Greenplum Database    Databases    Datatypes    SERIAL   
  138. "Some Blog Post" or, How I Learned to Stop Worrying and Like Red Junit Tests Tips and tricks for writing tests that fail well. What to mock, what to name your tests, and how to when. Categories:   Java    Testing   
  139. Building machine learning models at scale for data parallel problems on Pivotal's MPP databases Building machine learning models (ex: scikit-learn) at scale for data parallel problems on Pivotal’s MPP databases (Greenplum/HAWQ). Categories:   Data Science    Greenplum    Procedural Languages    Python   
  140. Making A Useful C++ Buildpack A useful C++ buildpack needs to consider header files and libraries, not just make. Here’s a story about how I made a useful buildpack for a C++ web framework. Categories:   CF Runtime   
  141. Algebraic Data Types In Kotlin Getting feedback quickly about mistakes in your code is a key tenet of agile development. This article will show you how to use algebraic data types and the Kotlin compiler to get fast feedback when you have missed handling an outcome for a business use case. Categories:   Kotlin    functional programming   
  142. The Journey of a Spring Boot application from Java 8 to Kotlin, part 3: Data Classes Kotlin data classes reduce a lot of boilerplate code when it comes to writing POJOs that are used for data exchange. Categories:   Spring Boot    Kotlin    Java   
  143. Current TransactionID in Greenplum Database How to find out the current TransactionID in Greenplum Database Categories:   PostgreSQL    Greenplum Database    Databases   
  144. The Journey of a Spring Boot application from Java 8 to Kotlin, part 2: Configuration Classes What do Spring Boot configuration classes look like in Kotlin? Categories:   Spring Boot    Kotlin   
  145. PgConf.Russia 2016 PgConf.Russia 2016 – Talk: How we made Greenplum Open Source Categories:   PostgreSQL    Conference   
  146. Exploring at Pivotal A candid insight into the adoption of the exploratory testing practice at Pivotal Labs. Categories:   Agile    Exploratory Testing    Charter    CF Runtime   
  147. The Journey of a Spring Boot application from Java 8 to Kotlin: The Application Class The first steps along the path of converting a fully functional Java 8/Spring Boot/Spring Cloud application to Kotlin. Categories:   Spring Boot    Kotlin   
  148. Capturing Network Traffic With Docker Containers How to capture and log internet traffic from programs using Docker containers. Categories:   Containers    Docker    Network Traffic Monitoring    Logging & Metrics   
  149. PostgreSQL Meetup in Berlin, 2016-01-26 Pivotal hosted a PostgreSQL Meetup in Berlin. Speakers: Andres Freund and Oleksandr Shulgin. Categories:   PostgreSQL    Meetup   
  150. Pivotal Data Open Source in 2016: community, community, community! When it comes to Open Source, Pivotal had one kick ass of a year in 2015. Here’s a sneak peak for 2016. Categories:   Open Source    Community    Big Data    Databases   
  151. GPORCA, A Modular Query Optimizer, Is Now Open-Source GPORCA has achieved an overall 5X performance improvement across all 99 industry standard benchmark queries. Now we call on the community to help take the project to the next level. Categories:   Big Data    Databases    Query Optimization    SQL   
  152. Pairing for Data Scientists Lets see how pair programming fits in the data science world. Categories:   Data Science    Pair Programming    Agile   
  153. Concourse Web Logging You need to debug your Concourse ATC server. How do you turn up the logging level to allow that? Categories:   Concourse   
  154. Deploying your first .NET app on Cloud Foundry PCF 1.6 brings with it support for .NET. Here’s how to get started. Categories:   .NET    CF Runtime   
  155. Intro to the Patch Command Quick intro on how to use the patch command to edit, and revert, the text of multiple files. Categories:   Patch    Golang   
  156. Abstraction, or, The Gift of Our Weak Brains Our brains are naturally limited. This can be a curse, or it can be a gift, depending on how you look at it. Categories:   Agile    Humans   
  157. Setting up Kotlin with Android and tests First impressions of Kotlin Categories:   Kotlin    Android   
  158. HTTP Trailers Signaling failure during an HTTP stream Categories:   Golang    HTTP    MySQL   
  159. The World's Smallest Concourse CI Server How to deploy a publicly-accessible, extremely lean Concourse CI server. Categories:   BOSH    Concourse   
  160. Using the Cloud Foundry Firehose Plugin Get your Cloud Foundry Firehose logs and metrics straight to your fingertips. Categories:   Firehose    Loggregator    CF CLI    Logging & Metrics   
  161. Agile and Program Logic On some of the differences and similarities in perspective between Agile/TDD programmers and developers of program-logic tools. Categories:   Agile   
  162. Scaling up to 2000 vms with BOSH In order to know if we can deploy 2000 vms with BOSH, we did a scaling test and this blog post list how we did it and the caveats we found. Categories:   BOSH   
  163. A Team Sport Welcome to our new Engineering Journal!
  164. Hanging by a Thread It is late on a Friday afternoon, and your web application has stopped responding to requests. The server is still reachable, and the Apache Tomcat process is still running–there are no errors in the logs. You want to go home but you can’t until it is fixed. What do you do? Categories:   Tomcat    Troubleshooting    Migrated Content   
  165. Integrating Jenkins and Apache Tomcat for Continuous Deployment The practice of automated continuous deployment ensures that the latest checked in code is deployed, running, and accessible to various roles within an organization. You can start practicing continuous deployment very quickly using Tomcat or tc server, Jenkins, and your source control system of choice. Categories:   Tomcat    Continuous Integration    Migrated Content   
  166. JVM Tuning for Apache Tomcat Performance Tuning the JVM for Running Apache Tomcat Categories:   Tomcat    Performance    Migrated Content   
  167. Apache Tomcat GC Measurement Setting Up Measurement of Garbage Collection in Apache Tomcat Categories:   Tomcat    Performance    Migrated Content   
  168. Session Fixation Protection An overview of session fixation attacks and how they are prevented in Apache Tomcat. Categories:   Tomcat    Security    Migrated Context   
  169. Apache Tomcat jdbc-pool Configuration and use of Apache Tomcat’s high concurrency database connection pool Categories:   Tomcat    Database    JDBC Pool    Migrated Content